Nov 03 17:30:56.761254 ip-10-0-139-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 17:30:56.761269 ip-10-0-139-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 17:30:56.761279 ip-10-0-139-156 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 17:30:56.761647 ip-10-0-139-156 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 17:31:06.850348 ip-10-0-139-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 17:31:06.850364 ip-10-0-139-156 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c1774ac95844480f90e24ded4c89403f --
Nov 03 17:33:10.725118 ip-10-0-139-156 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:33:11.129361 ip-10-0-139-156 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:33:11.129361 ip-10-0-139-156 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:33:11.129361 ip-10-0-139-156 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:33:11.129361 ip-10-0-139-156 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:33:11.134247 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134151    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.134247 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134242    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.134247 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134247    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.134247 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134251    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.134247 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134255    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.134247 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134260    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134264    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134268    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134272    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134276    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134280    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134289    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134293    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134297    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134300    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134304    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134308    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134312    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134315    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134319    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134322    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134326    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134330    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134333    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.134591 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134337    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134343    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134349    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134353    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134364    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134368    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134372    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134376    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134380    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134384    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134389    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134393    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134398    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134402    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134407    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134412    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134416    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134421    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134425    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.135410 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134429    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134434    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134439    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134443    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134447    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134451    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134488    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134495    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134500    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134505    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134509    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134514    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134518    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134523    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134526    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134531    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134535    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134539    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134544    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.136230 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134548    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134553    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134559    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134563    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134568    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134572    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134578    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134582    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134586    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134590    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134594    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134599    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134603    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134609    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134613    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134618    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134622    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134626    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134630    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134635    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.137040 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134639    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134643    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134648    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134652    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134668    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134672    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134679    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134685    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134691    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134696    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134701    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134706    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134710    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134714    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134718    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134723    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134744    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134749    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134754    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134758    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.137893 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134763    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134770    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.134774    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136115    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136126    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136131    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136136    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136140    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136144    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136148    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136153    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136158    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136163    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136167    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136172    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136176    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136180    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136188    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136193    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.138964 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136197    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136201    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136205    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136210    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136214    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136218    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136222    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136225    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136234    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136238    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136243    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136247    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136251    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136256    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136260    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136267    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136273    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136278    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136283    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.139860 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136287    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136292    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136296    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136300    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136304    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136308    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136313    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136317    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136321    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136326    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136330    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136334    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136338    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136342    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136347    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136351    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136355    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136359    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136363    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.140555 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136369    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136374    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136378    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136383    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136387    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136392    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136396    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136400    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136404    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136408    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136413    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136417    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136421    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136425    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136430    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136435    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136439    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136443    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136447    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136451    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.141181 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136456    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136460    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136464    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136467    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136475    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136481    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136486    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136490    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136494    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136498    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136502    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136506    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136510    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136515    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136519    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136524    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136530    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136534    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136539    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.141829 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136543    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136547    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136553    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136557    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136561    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136565    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136569    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136573    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136577    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136581    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136585    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.136589    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137448    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137458    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137463    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137467    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137472    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137477    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137482    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137486    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.142284 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137491    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137495    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137499    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137503    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137508    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137512    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137516    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137520    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137525    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137530    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137534    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137538    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137542    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137547    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137551    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137555    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137559    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137564    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137568    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.142874 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137572    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137576    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137580    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137585    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137589    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137593    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137597    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137602    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137606    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137610    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137615    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137619    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137623    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137627    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137631    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137636    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137640    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137644    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137648    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137653    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.143331 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137657    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137660    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137664    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137669    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137674    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137678    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137682    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137686    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137690    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137694    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137698    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137704    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137710    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137715    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137721    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137725    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137750    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137757    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.143866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137761    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137766    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137771    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137776    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137781    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137785    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137789    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137793    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137798    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137802    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137806    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137810    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137814    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137818    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137823    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137827    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137831    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137836    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137841    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137845    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.144326 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137849    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137853    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137857    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137861    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137866    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137871    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137875    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137879    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137883    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137887    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137893    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137897    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137901    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137905    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137909    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137913    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137917    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137922    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137926    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137931    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.144816 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.137937    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138090    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138103    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138113    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138120    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138127    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138133    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138140    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138147    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138152    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138156    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138163    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138169    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138175    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138179    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138184    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138189    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138194    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138198    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138205    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138210    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138215    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138223    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138229    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:33:11.145297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138235    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138240    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138245    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138251    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138256    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138261    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138266    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138271    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138276    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138283    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138288    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138293    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138297    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138303    2579 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138308    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138315    2579 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138320    2579 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138324    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138329    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138334    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138342    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138347    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138352    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138357    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138362    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:33:11.145965 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138366    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138371    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138375    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138380    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138385    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138391    2579 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138398    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138403    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138408    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138413    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138419    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138424    2579 flags.go:64] FLAG: --help="false"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138429    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138434    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138439    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138444    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138449    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138456    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138461    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138466    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138470    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138474    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138479    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:33:11.146578 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138484    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138489    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138494    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138498    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138504    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138509    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138514    2579 flags.go:64] FLAG: --lock-file=""
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138519    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138524    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138528    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138537    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138542    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138547    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138552    2579 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138558    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138564    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138568    2579 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138573    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138581    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138588    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138595    2579 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138600    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138604    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138609    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138614    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:33:11.147451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138618    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138623    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138628    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138640    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138645    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138650    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138655    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138660    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138664    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138669    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138674    2579 flags.go:64] FLAG: --port="10250"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138678    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138682    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0566d144036bbe524"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138687    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138692    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138696    2579 flags.go:64] FLAG: --register-node="true"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138701    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138706    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138711    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138715    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138719    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138745    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138750    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138756    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138760    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138765    2579 flags.go:64] FLAG: --runonce="false"
Nov 03 17:33:11.148226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138771    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138775    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138780    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138784    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138789    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138794    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138799    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138803    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138807    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138812    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138817    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138821    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138825    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138830    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138835    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138844    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138848    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138852    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138860    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138864    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138876    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138880    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138885    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138889    2579 flags.go:64] FLAG: --v="2"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138897    2579 flags.go:64] FLAG: --version="false"
Nov 03 17:33:11.148994 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138907    2579 flags.go:64] FLAG: --vmodule=""
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138913    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.138920    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139139    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139147    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139154    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139158    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139168    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139172    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139176    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139181    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139185    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139189    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139193    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139197    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139202    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139205    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139210    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139214    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139218    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.149662 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139223    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139226    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139231    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139235    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139239    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139244    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139248    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139253    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139258    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139262    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139266    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139270    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139275    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139279    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139284    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139288    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139293    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139297    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139301    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.150283 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139306    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139311    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139315    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139319    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139323    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139328    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139332    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139337    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139341    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139345    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139349    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139354    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139358    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139362    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139366    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139371    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139375    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139380    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139384    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139389    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.150794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139393    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139397    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139403    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139407    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139411    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139418    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139424    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139432    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139436    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139441    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139445    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139449    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139454    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139459    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139463    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139468    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139472    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139478    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.151292 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139484    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139488    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139493    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139498    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139502    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139506    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139510    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139515    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139519    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139523    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139527    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139531    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139535    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139539    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139544    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139548    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139552    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139557    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139561    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.151765 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139565    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139569    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139576    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139580    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139584    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139589    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139593    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139599    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139604    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139608    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139612    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139617    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139621    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.139662    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139718    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139725    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.152229 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139750    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139762    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139766    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139771    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139775    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139779    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139783    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139787    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139792    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139796    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139800    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139804    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139808    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139812    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139816    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139821    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139826    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139830    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139835    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.152610 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139841    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139845    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139849    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139854    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139860    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139864    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139868    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139872    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139876    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139880    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139885    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139889    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139894    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139898    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139902    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139906    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139911    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139915    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139919    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139924    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.153093 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139928    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139933    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139937    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139941    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139946    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139950    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139954    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139958    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139962    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139968    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139972    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139976    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139982    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139987    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139991    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.139996    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140006    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140014    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140018    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140023    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.153592 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140028    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140033    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140037    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140041    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140045    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140050    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140054    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140058    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140065    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140072    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140077    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140081    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140085    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140090    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140094    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140098    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140102    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140107    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140111    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.154224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140115    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140119    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140123    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140128    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140132    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140138    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140142    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140146    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140150    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140156    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140161    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140165    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140169    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140173    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140177    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140181    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140186    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140190    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140195    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140199    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.154713 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140203    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140207    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140212    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140216    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140221    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.140226    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.148951    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.148967    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.150233    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.150240    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150302    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150308    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150311    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150315    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150318    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150321    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150324    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150326    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150329    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150332    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.155232 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150334    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150337    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150339    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150342    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150344    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150348    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150350    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150353    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150355    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150358    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150361    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150364    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150366    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150369    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150372    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150374    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150377    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150380    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150383    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150385    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.155701 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150388    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150390    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150393    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150395    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150398    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150401    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150404    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150406    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150408    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150411    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150413    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150417    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150419    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150422    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150424    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150427    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150431    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150433    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150435    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150438    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.156224 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150440    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150443    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150446    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150449    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150451    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150454    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150456    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150459    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150461    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150464    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150467    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150470    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150472    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150475    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150477    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150480    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150482    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150485    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150488    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150491    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.156703 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150493    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150495    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150498    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150500    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150503    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150505    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150509    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150514    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150517    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150521    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150524    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150527    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150530    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150533    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150536    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150538    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150542    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150545    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150548    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.157199 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150552    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150555    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150558    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150561    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150564    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150567    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150569    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150572    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150574    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150577    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150579    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150582    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150584    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150587    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150589    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150592    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.157645 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150594    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.150628    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150797    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150804    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150808    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150810    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150813    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150817    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150820    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150822    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150825    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150828    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150830    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150833    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150835    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.158039 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150838    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150840    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150843    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150846    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150849    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150852    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150855    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150857    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150860    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150862    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150866    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150868    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150871    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150874    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150876    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150879    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150881    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150884    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150886    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.158402 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150888    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150891    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150893    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150896    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150898    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150900    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150904    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150907    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150909    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150912    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150915    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150917    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150920    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150922    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150925    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150927    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150930    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150932    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150935    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150937    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.158911 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150939    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150942    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150944    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150946    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150949    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150951    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150953    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150956    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150958    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150960    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150963    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150965    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150968    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150970    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150972    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150975    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150977    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150979    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150982    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.159387 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150985    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150988    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150991    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150996    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.150999    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151002    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151005    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151007    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151010    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151012    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151015    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151018    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151020    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151023    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151025    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151027    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151031    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151034    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151037    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.159866 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151039    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151042    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151044    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151046    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151049    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151052    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151054    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151057    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151059    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151062    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151064    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151066    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151069    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151071    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151075    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151078    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.160318 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.151102    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151159    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151164    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151167    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151170    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151175    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151179    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151182    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151185    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151188    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151191    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151193    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151196    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151199    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151201    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151204    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151206    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151209    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151211    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:11.160686 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151213    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151216    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151218    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151221    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151223    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151226    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151228    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151230    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151233    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151235    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151238    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151240    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151244    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151246    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151249    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151251    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151254    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151256    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:11.161202 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151261    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151263    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151266    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151268    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151271    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151273    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151276    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151279    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151282    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151287    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151289    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151293    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151295    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151298    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151301    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151303    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151306    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151308    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151311    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151313    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:11.161636 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151316    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151318    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151320    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151323    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151326    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151328    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151331    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151334    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151337    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151339    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151342    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151344    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151346    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151349    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151351    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151354    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151356    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151358    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151361    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151363    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:11.162124 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151365    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151368    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151371    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151374    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151376    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151379    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151381    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151383    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151386    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151389    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151393    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151395    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151398    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151400    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151402    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151405    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151407    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151410    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151412    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:11.162622 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151416    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151420    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151422    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151425    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151427    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151430    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151432    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151434    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151437    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:11.151440    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.151501    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.153540    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.154457    2579 server.go:1014] "Starting client certificate rotation"
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.154553    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.155066    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.156565    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:33:11.163080 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.158090    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:33:11.169024 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.169007    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:33:11.172602 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.172585    2579 log.go:25] "Validated CRI v1 image API"
Nov 03 17:33:11.173451 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.173429    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:33:11.176605 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.176576    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9af7367f-55c7-4331-ab17-3d39fc8e8584:/dev/nvme0n1p3 9fbbc0f0-f8a8-418c-9998-5caa8607dc0c:/dev/nvme0n1p4]
Nov 03 17:33:11.176656 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.176605    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 17:33:11.181664 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.181549    2579 manager.go:217] Machine: {Timestamp:2026-11-03 17:33:11.180268162 +0000 UTC m=+0.325735746 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105799 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279ce56d4e03e21aa1eeb43fe4790a SystemUUID:ec279ce5-6d4e-03e2-1aa1-eeb43fe4790a BootID:c1774ac9-5844-480f-90e2-4ded4c89403f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:c6:3b:56:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:c6:3b:56:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:7a:c8:9c:96:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:33:11.182141 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.182131    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:33:11.182236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.182222    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:33:11.183288 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.183261    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:33:11.183426 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.183290    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-156.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:33:11.183470 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.183429    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:33:11.183470 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.183437    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:33:11.183470 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.183466    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:33:11.183541 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.183483    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:33:11.184224 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.184213    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:33:11.187880 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.187866    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:33:11.188006 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.187997    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:33:11.188784 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.188774    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:33:11.188815 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.188789    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:33:11.188815 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.188805    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:33:11.189286 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.189256    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:33:11.189385 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.189305    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:33:11.189812 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.189798    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:33:11.193115 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.193095    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:33:11.197031 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197018    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:33:11.197680 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197668    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:33:11.197721 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197699    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:33:11.197869 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197859    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:33:11.197922 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197874    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:33:11.197922 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197881    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:33:11.197922 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197886    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:33:11.197922 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197892    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:33:11.197922 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.197897    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:33:11.198475 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.198467    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:33:11.198503 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.198476    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:33:11.198503 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.198483    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:33:11.198503 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.198488    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:33:11.198503 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.198497    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:33:11.198622 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.198506    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:33:11.199803 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.199793    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:33:11.199803 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.199803    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:33:11.203660 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.203644    2579 server.go:1263] "Started kubelet"
Nov 03 17:33:11.203822 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.203777    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:33:11.203890 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.203843    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:33:11.204191 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.204142    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:33:11.204518 ip-10-0-139-156 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:33:11.205112 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.204839    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:33:11.206893 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.206870    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:33:11.208290 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.208271    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 17:33:11.209672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.209652    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:33:11.209672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.209664    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:33:11.209818 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.209718    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-156.us-west-1.compute.internal\" not found"
Nov 03 17:33:11.210230 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210199    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:33:11.210326 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210237    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:33:11.210409 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210393    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:33:11.210409 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210410    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:33:11.210490 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210461    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:33:11.210700 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210685    2579 factory.go:55] Registering systemd factory
Nov 03 17:33:11.210783 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210704    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:33:11.210938 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210925    2579 factory.go:153] Registering CRI-O factory
Nov 03 17:33:11.211001 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210941    2579 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:33:11.211001 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.210989    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:33:11.211095 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.211011    2579 factory.go:103] Registering Raw factory
Nov 03 17:33:11.211095 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.211026    2579 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:33:11.211978 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.211955    2579 manager.go:319] Starting recovery of all containers
Nov 03 17:33:11.224986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.224959    2579 manager.go:324] Recovery completed
Nov 03 17:33:11.226410 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.226392    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 17:33:11.229238 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.229226    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.230991 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.230965    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.231038 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.231017    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.231038 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.231032    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.231628 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.231612    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:33:11.231628 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.231623    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.231770 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.231624    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:33:11.231770 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.231686    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:33:11.232335 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.232320    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.232401 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.232347    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.232401 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.232359    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.233906 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.233893    2579 policy_none.go:50] "Start"
Nov 03 17:33:11.233951 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.233910    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:33:11.233951 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.233920    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:33:11.235919 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.235908    2579 policy_none.go:44] "Start"
Nov 03 17:33:11.271572 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.271532    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:33:11.275627 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.275456    2579 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:33:11.275692 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.275646    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:33:11.275692 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.275656    2579 server.go:87] "Starting device plugin registration server"
Nov 03 17:33:11.275903 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.275892    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:33:11.275943 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.275904    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:33:11.276042 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.276022    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:33:11.276226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.276129    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:33:11.276226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.276140    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:33:11.276775 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.276758    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:33:11.276851 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.276807    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-156.us-west-1.compute.internal\" not found"
Nov 03 17:33:11.329128 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.329090    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:33:11.331101 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.330509    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:33:11.331101 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.330538    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:33:11.331101 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.330565    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:33:11.331101 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.330603    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 17:33:11.352672 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.352642    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-156.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 17:33:11.353429 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.352643    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-156.us-west-1.compute.internal.18e4996077a68e77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-156.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-156.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:33:11.203614327 +0000 UTC m=+0.349081909,LastTimestamp:2026-11-03 17:33:11.203614327 +0000 UTC m=+0.349081909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-156.us-west-1.compute.internal,}"
Nov 03 17:33:11.376977 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.376949    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.377910 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.377888    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.378019 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.377927    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.378019 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.377938    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.378019 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.377967    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.401245 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.401171    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-156.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 17:33:11.431602 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.431556    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"]
Nov 03 17:33:11.431793 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.431638    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.432831 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.432807    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.432914 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.432840    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.432914 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.432853    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.434037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.434013    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:33:11.434140 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.434064    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ws2b6"
Nov 03 17:33:11.435078 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435064    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.435212 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.435259 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435229    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.435951 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435930    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.436037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435964    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.436037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435978    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.436037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.435938    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.436037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.436015    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.436037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.436025    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.438618 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.438604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.438705 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.438628    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:11.439346 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.439331    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:11.439428 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.439364    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:11.439428 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.439378    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:11.458405 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.458384    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-156.us-west-1.compute.internal\" not found" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.460846 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.460826    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-156.us-west-1.compute.internal\" not found" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.512380 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.512345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.512380 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.512382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.512573 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.512405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/779eab1b0b29ccc85d3654a5ae82e6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"779eab1b0b29ccc85d3654a5ae82e6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.514530 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.514509    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:11.516198 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.516182    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:11.613118 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.613083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.613272 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.613148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.613272 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.613179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/779eab1b0b29ccc85d3654a5ae82e6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"779eab1b0b29ccc85d3654a5ae82e6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.613272 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.613207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.613272 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.613227    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/779eab1b0b29ccc85d3654a5ae82e6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"779eab1b0b29ccc85d3654a5ae82e6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.613272 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.613260    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.614457 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.614437    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:11.615334 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.615303    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.615334 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:11.615332    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-156.us-west-1.compute.internal\": node \"ip-10-0-139-156.us-west-1.compute.internal\" not found"
Nov 03 17:33:11.615513 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.615453    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:11.710290 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.710202    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.761533 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.761496    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.764424 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.764402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.773440 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.773416    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:11.773563 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.773533    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:33:11.832259 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:11.832229    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:12.154339 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.154254    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 17:33:12.154898 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.154474    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:33:12.154898 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.154476    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:33:12.154898 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.154441    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa49546cbf5c9454088617b8e9f7adf8-66509c371b41a33e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-156.us-west-1.compute.internal.18e499607949024b\": read tcp 10.0.139.156:58688->3.149.82.99:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-156.us-west-1.compute.internal.18e499607949024b  default   5291 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-156.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-156.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-156.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:33:11 +0000 UTC,LastTimestamp:2026-11-03 17:33:11.377941643 +0000 UTC m=+0.523409226,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-156.us-west-1.compute.internal,}"
Nov 03 17:33:12.154898 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.154476    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:33:12.190387 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.190359    2579 apiserver.go:52] "Watching apiserver"
Nov 03 17:33:12.210442 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.210417    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 17:33:12.438261 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.438155    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:28:11 +0000 UTC" deadline="2028-05-07 03:11:39.740462846 +0000 UTC"
Nov 03 17:33:12.438261 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.438204    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13209h38m27.302263981s"
Nov 03 17:33:12.462701 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.462669    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:33:12.463663 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.463626    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-4tnrc","openshift-multus/network-metrics-daemon-8vqsh","openshift-network-diagnostics/network-check-target-tvv9c","openshift-network-operator/iptables-alerter-sdjkq","openshift-ovn-kubernetes/ovnkube-node-hwxhd","kube-system/global-pull-secret-syncer-cz52j","openshift-multus/multus-rxbv7","openshift-cluster-node-tuning-operator/tuned-7xt6l","openshift-image-registry/node-ca-kpf92","kube-system/konnectivity-agent-hxznn","kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v","openshift-dns/node-resolver-bs2m4"]
Nov 03 17:33:12.467323 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.467299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.469382 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.469364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.471453 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.471439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.473505 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.473488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:12.473562 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.473545    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:12.475661 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.475645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.477689 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.477675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.477759 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.477741    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:12.479782 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.479768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:12.479847 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.479829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:12.481878 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.481866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.484050 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.484035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.486336 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.486319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.488608 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.488594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.490788 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.490772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.511385 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.511369    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:33:12.516606 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-cnibin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.516672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-systemd\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.516672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-var-lib-kubelet\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.516672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-tmp\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.516672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-etc-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.516885 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-bin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.516885 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.516885 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5j5\" (UniqueName: \"kubernetes.io/projected/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kube-api-access-xq5j5\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.516885 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-var-lib-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nv6s\" (UniqueName: \"kubernetes.io/projected/9e659c14-b9ea-4485-a903-eb9ed8e42272-kube-api-access-6nv6s\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.517035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-conf\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.517035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b47b5da-225b-4e79-aab7-f81b12aa6191-host\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.517035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-registration-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.517035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.516986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-ovn\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517216 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:12.517216 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-dbus\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.517216 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-script-lib\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517216 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-netns\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517216 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-kubelet\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517216 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-hostroot\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517219    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-lib-modules\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-tuned\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517272    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkp8p"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhm9\" (UniqueName: \"kubernetes.io/projected/1df9489d-1992-48e2-9fca-b6a83d7a0cff-kube-api-access-ddhm9\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.517381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-k8s-cni-cncf-io\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517593 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4951c9-ab34-4727-9272-fc331a7fae42-konnectivity-ca\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.517593 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-device-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.517593 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.517593 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-multus-certs\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517593 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-sys-fs\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.517779 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-multus\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517779 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.517779 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-log-socket\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517779 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-cni-binary-copy\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b47b5da-225b-4e79-aab7-f81b12aa6191-serviceca\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb54bc0-f60d-4745-bcfe-5978119c0bde-hosts-file\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4rf\" (UniqueName: \"kubernetes.io/projected/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-kube-api-access-jc4rf\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-host-slash\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-kubelet-config\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4951c9-ab34-4727-9272-fc331a7fae42-agent-certs\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.517920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dgmd\" (UniqueName: \"kubernetes.io/projected/dbb54bc0-f60d-4745-bcfe-5978119c0bde-kube-api-access-2dgmd\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-config\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4p6\" (UniqueName: \"kubernetes.io/projected/7f48d878-d642-49c6-a4db-6e74ea2542fc-kube-api-access-8d4p6\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.517995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb54bc0-f60d-4745-bcfe-5978119c0bde-tmp-dir\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-node-log\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-env-overrides\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-kubernetes\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-sys\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-systemd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-cnibin\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-system-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-host\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-slash\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-system-cni-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysconfig\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-run\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bqd\" (UniqueName: \"kubernetes.io/projected/4b47b5da-225b-4e79-aab7-f81b12aa6191-kube-api-access-r9bqd\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.518528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-socket-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-kubelet\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-bin\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-conf-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-daemon-config\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwgl\" (UniqueName: \"kubernetes.io/projected/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-kube-api-access-jlwgl\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/634768ef-d8f7-474d-9ee4-b479a1b1f17b-kube-api-access-btmj5\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovn-node-metrics-cert\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-socket-dir-parent\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-etc-kubernetes\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-iptables-alerter-script\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-modprobe-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-systemd-units\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-netns\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.518925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.519392 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.518941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-netd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.523254 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523180    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.523254 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523222    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.523254 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523226    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:33:12.523523 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523490    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:33:12.523640 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523540    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:33:12.523640 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523560    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:33:12.523789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523678    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:33:12.523789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523682    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:33:12.523789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523693    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nx9c7\""
Nov 03 17:33:12.523789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523700    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.523789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523708    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.523789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523773    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:33:12.524120 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.523801    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xxxqk\""
Nov 03 17:33:12.524120 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.524008    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfxsl\""
Nov 03 17:33:12.565195 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.565176    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:33:12.619436 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-systemd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.619436 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-cnibin\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-system-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-host\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619527    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-cnibin\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-systemd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619583    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619568    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-system-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-slash\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619613    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-slash\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-system-cni-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619628    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.619652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619555    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-host\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619670    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-system-cni-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysconfig\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619763    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysconfig\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619771    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-run\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619802    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-run\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bqd\" (UniqueName: \"kubernetes.io/projected/4b47b5da-225b-4e79-aab7-f81b12aa6191-kube-api-access-r9bqd\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619834    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-socket-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-kubelet\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-bin\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-conf-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-daemon-config\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619954    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-socket-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwgl\" (UniqueName: \"kubernetes.io/projected/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-kube-api-access-jlwgl\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.620236 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619958    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-kubelet\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-conf-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619998    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-bin\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.619995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/634768ef-d8f7-474d-9ee4-b479a1b1f17b-kube-api-access-btmj5\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovn-node-metrics-cert\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-socket-dir-parent\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-etc-kubernetes\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-iptables-alerter-script\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-socket-dir-parent\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-modprobe-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-etc-kubernetes\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-systemd-units\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-netns\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-netd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620378    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-modprobe-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-systemd-units\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-cnibin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-netns\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-systemd\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620437    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-cnibin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620453    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-netd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620466    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-systemd\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-var-lib-kubelet\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620480    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-tmp\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620496    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-daemon-config\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-etc-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620513    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-var-lib-kubelet\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-bin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5j5\" (UniqueName: \"kubernetes.io/projected/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kube-api-access-xq5j5\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-etc-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.621986 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620575    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-bin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-var-lib-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv6s\" (UniqueName: \"kubernetes.io/projected/9e659c14-b9ea-4485-a903-eb9ed8e42272-kube-api-access-6nv6s\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-conf\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b47b5da-225b-4e79-aab7-f81b12aa6191-host\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-registration-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620787    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-var-lib-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-ovn\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620825    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b47b5da-225b-4e79-aab7-f81b12aa6191-host\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-dbus\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-ovn\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-conf\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-script-lib\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.622830 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-registration-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-dbus\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.620985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-netns\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-kubelet\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-hostroot\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-lib-modules\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-netns\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-tuned\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhm9\" (UniqueName: \"kubernetes.io/projected/1df9489d-1992-48e2-9fca-b6a83d7a0cff-kube-api-access-ddhm9\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621088    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-kubelet\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621128    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-lib-modules\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621092    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-hostroot\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621127    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-k8s-cni-cncf-io\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.623590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4951c9-ab34-4727-9272-fc331a7fae42-konnectivity-ca\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-device-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-multus-certs\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-sys-fs\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-multus\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621379    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-device-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-log-socket\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-cni-binary-copy\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-k8s-cni-cncf-io\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b47b5da-225b-4e79-aab7-f81b12aa6191-serviceca\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb54bc0-f60d-4745-bcfe-5978119c0bde-hosts-file\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4rf\" (UniqueName: \"kubernetes.io/projected/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-kube-api-access-jc4rf\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-host-slash\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-kubelet-config\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4951c9-ab34-4727-9272-fc331a7fae42-agent-certs\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.624412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgmd\" (UniqueName: \"kubernetes.io/projected/dbb54bc0-f60d-4745-bcfe-5978119c0bde-kube-api-access-2dgmd\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-config\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4p6\" (UniqueName: \"kubernetes.io/projected/7f48d878-d642-49c6-a4db-6e74ea2542fc-kube-api-access-8d4p6\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb54bc0-f60d-4745-bcfe-5978119c0bde-hosts-file\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb54bc0-f60d-4745-bcfe-5978119c0bde-tmp-dir\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621756    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-multus-certs\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-node-log\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-env-overrides\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-node-log\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-kubernetes\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-sys\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.621955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-sys\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622041    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-kubelet-config\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622124    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-sys-fs\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622237    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-multus\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.622250    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:12.624923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622290    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-kubernetes\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.622327    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:33:13.122306171 +0000 UTC m=+2.267773742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-log-socket\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622380    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-host-slash\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622610    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-cni-binary-copy\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622649    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb54bc0-f60d-4745-bcfe-5978119c0bde-tmp-dir\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.622671    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.622715    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:33:13.122704518 +0000 UTC m=+2.268172101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622777    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-script-lib\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.622991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-env-overrides\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623060    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-config\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623533    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623588    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623645    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623704    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623778    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623810    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623945    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623953    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.623991    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.625393 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.624502    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-tuned\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.626020 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.624514    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-tmp\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.626020 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.624540    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovn-node-metrics-cert\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.626020 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.625790    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:33:12.632616 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.632598    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4951c9-ab34-4727-9272-fc331a7fae42-konnectivity-ca\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.632896 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.632882    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b47b5da-225b-4e79-aab7-f81b12aa6191-serviceca\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.651058 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651035    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2vfjt\""
Nov 03 17:33:12.651172 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651132    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xhj6\""
Nov 03 17:33:12.651172 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651151    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:33:12.651172 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651035    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:33:12.651359 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651201    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:12.651359 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651221    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:33:12.651359 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651269    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jxmj\""
Nov 03 17:33:12.651359 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651273    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxb7s\""
Nov 03 17:33:12.651359 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651305    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:33:12.651499 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651357    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tvvh\""
Nov 03 17:33:12.651499 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651398    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qfzl\""
Nov 03 17:33:12.651981 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.651963    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-iptables-alerter-script\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.654470 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.654451    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4951c9-ab34-4727-9272-fc331a7fae42-agent-certs\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.681626 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.681597    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:12.681626 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.681627    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:12.681882 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.681641    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:12.681882 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.681770    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:33:13.181747627 +0000 UTC m=+2.327215215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:12.683075 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.682948    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwgl\" (UniqueName: \"kubernetes.io/projected/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-kube-api-access-jlwgl\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.683430 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.683409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/634768ef-d8f7-474d-9ee4-b479a1b1f17b-kube-api-access-btmj5\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:12.683865 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.683841    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bqd\" (UniqueName: \"kubernetes.io/projected/4b47b5da-225b-4e79-aab7-f81b12aa6191-kube-api-access-r9bqd\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.684374 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.684352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv6s\" (UniqueName: \"kubernetes.io/projected/9e659c14-b9ea-4485-a903-eb9ed8e42272-kube-api-access-6nv6s\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.684517 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.684501    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5j5\" (UniqueName: \"kubernetes.io/projected/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kube-api-access-xq5j5\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.684968 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.684946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgmd\" (UniqueName: \"kubernetes.io/projected/dbb54bc0-f60d-4745-bcfe-5978119c0bde-kube-api-access-2dgmd\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.685217 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.685197    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4p6\" (UniqueName: \"kubernetes.io/projected/7f48d878-d642-49c6-a4db-6e74ea2542fc-kube-api-access-8d4p6\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.685255 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.685197    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhm9\" (UniqueName: \"kubernetes.io/projected/1df9489d-1992-48e2-9fca-b6a83d7a0cff-kube-api-access-ddhm9\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.686222 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.686202    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4rf\" (UniqueName: \"kubernetes.io/projected/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-kube-api-access-jc4rf\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.808069 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.807982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:12.823445 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.823418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:33:12.831268 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.831153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rxbv7"
Nov 03 17:33:12.835808 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.835722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:33:12.842014 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.841971    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779eab1b0b29ccc85d3654a5ae82e6cd.slice/crio-f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634 WatchSource:0}: Error finding container f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634: Status 404 returned error can't find the container with id f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634
Nov 03 17:33:12.842931 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.842906    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5908cbf5_2e94_4a1c_9c81_f7adfd472da0.slice/crio-8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0 WatchSource:0}: Error finding container 8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0: Status 404 returned error can't find the container with id 8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0
Nov 03 17:33:12.847010 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.846982    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e659c14_b9ea_4485_a903_eb9ed8e42272.slice/crio-c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127 WatchSource:0}: Error finding container c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127: Status 404 returned error can't find the container with id c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127
Nov 03 17:33:12.848158 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.848102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:33:12.848276 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.848258    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:33:12.848902 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.848766    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f48d878_d642_49c6_a4db_6e74ea2542fc.slice/crio-cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342 WatchSource:0}: Error finding container cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342: Status 404 returned error can't find the container with id cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342
Nov 03 17:33:12.852616 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.852593    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6544fcbf_d2a3_48fc_8f79_71cc649dd82b.slice/crio-d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8 WatchSource:0}: Error finding container d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8: Status 404 returned error can't find the container with id d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8
Nov 03 17:33:12.853305 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.853276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:33:12.853544 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.853524    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fb4e1599f1bd3962469eeb7b079336.slice/crio-2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1 WatchSource:0}: Error finding container 2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1: Status 404 returned error can't find the container with id 2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1
Nov 03 17:33:12.858321 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.858299    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df9489d_1992_48e2_9fca_b6a83d7a0cff.slice/crio-b61c4d90239234f321c9f855ab45344a32cd01e57eb0bd0325a9f25e5eef4fe5 WatchSource:0}: Error finding container b61c4d90239234f321c9f855ab45344a32cd01e57eb0bd0325a9f25e5eef4fe5: Status 404 returned error can't find the container with id b61c4d90239234f321c9f855ab45344a32cd01e57eb0bd0325a9f25e5eef4fe5
Nov 03 17:33:12.862805 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.862781    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b47b5da_225b_4e79_aab7_f81b12aa6191.slice/crio-1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e WatchSource:0}: Error finding container 1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e: Status 404 returned error can't find the container with id 1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e
Nov 03 17:33:12.878928 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.878905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:12.887949 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.887923    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4951c9_ab34_4727_9272_fc331a7fae42.slice/crio-9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b WatchSource:0}: Error finding container 9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b: Status 404 returned error can't find the container with id 9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b
Nov 03 17:33:12.899588 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.899567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:33:12.901694 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:12.901675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:33:12.909398 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.909371    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2021b15f_bca7_48eb_9280_bbbeb6c7bf9d.slice/crio-d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b WatchSource:0}: Error finding container d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b: Status 404 returned error can't find the container with id d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b
Nov 03 17:33:12.911304 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:33:12.911277    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb54bc0_f60d_4745_bcfe_5978119c0bde.slice/crio-c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711 WatchSource:0}: Error finding container c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711: Status 404 returned error can't find the container with id c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711
Nov 03 17:33:12.913305 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.913284    2579 kuberuntime_manager.go:1707] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-bs2m4_openshift-dns(dbb54bc0-f60d-4745-bcfe-5978119c0bde): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Nov 03 17:33:12.913383 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:12.913317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-bs2m4" podUID="dbb54bc0-f60d-4745-bcfe-5978119c0bde"
Nov 03 17:33:13.125538 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.125499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:13.125538 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.125544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:13.125776 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.125658    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:13.125776 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.125727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:33:14.125712237 +0000 UTC m=+3.271179823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:13.125776 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.125660    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:13.125885 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.125790    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:33:14.125781967 +0000 UTC m=+3.271249539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:13.226138 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.226092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:13.226775 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.226264    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:13.226775 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.226280    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:13.226775 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.226290    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:13.226775 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.226348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:33:14.226330332 +0000 UTC m=+3.371797926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:13.272835 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.272803    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:13.335831 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.335708    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xt6l" event={"ID":"1df9489d-1992-48e2-9fca-b6a83d7a0cff","Type":"ContainerStarted","Data":"b61c4d90239234f321c9f855ab45344a32cd01e57eb0bd0325a9f25e5eef4fe5"}
Nov 03 17:33:13.336959 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.336933    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rxbv7" event={"ID":"7f48d878-d642-49c6-a4db-6e74ea2542fc","Type":"ContainerStarted","Data":"cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342"}
Nov 03 17:33:13.338037 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.338010    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerStarted","Data":"c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127"}
Nov 03 17:33:13.339147 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.339125    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0"}
Nov 03 17:33:13.340259 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.340229    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdjkq" event={"ID":"6544fcbf-d2a3-48fc-8f79-71cc649dd82b","Type":"ContainerStarted","Data":"d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8"}
Nov 03 17:33:13.341329 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.341307    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"00fb4e1599f1bd3962469eeb7b079336","Type":"ContainerStarted","Data":"2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1"}
Nov 03 17:33:13.343187 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.343160    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"779eab1b0b29ccc85d3654a5ae82e6cd","Type":"ContainerStarted","Data":"f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634"}
Nov 03 17:33:13.344969 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.344944    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bs2m4" event={"ID":"dbb54bc0-f60d-4745-bcfe-5978119c0bde","Type":"ContainerStarted","Data":"c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711"}
Nov 03 17:33:13.346190 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.346145    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v" event={"ID":"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d","Type":"ContainerStarted","Data":"d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b"}
Nov 03 17:33:13.346663 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:13.346640    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-bs2m4" podUID="dbb54bc0-f60d-4745-bcfe-5978119c0bde"
Nov 03 17:33:13.347202 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.347175    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hxznn" event={"ID":"7d4951c9-ab34-4727-9272-fc331a7fae42","Type":"ContainerStarted","Data":"9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b"}
Nov 03 17:33:13.348242 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.348224    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kpf92" event={"ID":"4b47b5da-225b-4e79-aab7-f81b12aa6191","Type":"ContainerStarted","Data":"1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e"}
Nov 03 17:33:13.392356 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.392322    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:13.440407 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.440372    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:13.520509 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.520441    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:12 +0000 UTC" deadline="2028-05-30 12:59:34.748419697 +0000 UTC"
Nov 03 17:33:13.520509 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:13.520495    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13771h26m21.227927233s"
Nov 03 17:33:14.133288 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.133191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:14.133288 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.133260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:14.133537 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.133372    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:14.133537 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.133387    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:14.133537 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.133449    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:33:16.133428293 +0000 UTC m=+5.278895881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:14.133537 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.133471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:33:16.133459161 +0000 UTC m=+5.278926747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:14.235018 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.234675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:14.235018 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.234872    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:14.235018 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.234897    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:14.235018 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.234912    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:14.235018 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.234977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:33:16.234957612 +0000 UTC m=+5.380425201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:14.331290 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.331245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:14.331496 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.331410    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:14.331575 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.331497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:14.331575 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.331563    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:14.331686 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.331614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:14.331734 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:14.331698    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:14.521013 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.520940    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:12 +0000 UTC" deadline="2028-05-07 21:40:06.448368395 +0000 UTC"
Nov 03 17:33:14.521013 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:14.521010    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13228h6m51.927363601s"
Nov 03 17:33:16.153291 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:16.153241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:16.153849 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:16.153352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:16.153849 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.153511    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:16.153849 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.153593    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:33:20.153571458 +0000 UTC m=+9.299039044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:16.153849 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.153676    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:16.153849 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.153713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:33:20.153701367 +0000 UTC m=+9.299168958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:16.254398 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:16.254347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:16.254614 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.254583    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:16.254614 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.254608    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:16.254720 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.254622    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:16.254720 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.254688    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:33:20.254667888 +0000 UTC m=+9.400135471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:16.331317 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:16.331285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:16.331614 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.331448    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:16.331614 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:16.331475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:16.331614 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.331599    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:16.331898 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:16.331638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:16.331898 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:16.331721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:18.331768 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.331449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:18.332267 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:18.331882    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:18.332267 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.331984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:18.332267 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:18.332066    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:18.332267 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.332126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:18.332267 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:18.332200    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:18.396502 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.396461    2579 generic.go:356] "Generic (PLEG): container finished" podID="00fb4e1599f1bd3962469eeb7b079336" containerID="0b1cc32e422080cdb1c7ec14f3ce7132ca668607d8b10791f5bf2509efef69d6" exitCode=0
Nov 03 17:33:18.396692 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.396556    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"00fb4e1599f1bd3962469eeb7b079336","Type":"ContainerDied","Data":"0b1cc32e422080cdb1c7ec14f3ce7132ca668607d8b10791f5bf2509efef69d6"}
Nov 03 17:33:18.399918 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.399882    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"779eab1b0b29ccc85d3654a5ae82e6cd","Type":"ContainerStarted","Data":"216f471864ddf9f1d51998626b09dd4ac94741bebe637244e21ec5405353eab4"}
Nov 03 17:33:18.410499 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.410461    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hxznn" event={"ID":"7d4951c9-ab34-4727-9272-fc331a7fae42","Type":"ContainerStarted","Data":"806614b276edea8fb247809492c8efd9a46a11e062f8af74f0d2e5e38b664c7e"}
Nov 03 17:33:18.414722 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.414620    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="810cd0824d66ec1eafdc0d9c1d54f0cf55d3edc8620a04eca17374a4110e870e" exitCode=0
Nov 03 17:33:18.414856 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.414720    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerDied","Data":"810cd0824d66ec1eafdc0d9c1d54f0cf55d3edc8620a04eca17374a4110e870e"}
Nov 03 17:33:18.653863 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.653769    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hxznn" podStartSLOduration=1.684089434 podStartE2EDuration="6.653748171s" totalImagesPullingTime="4.969658737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:18.65264969 +0000 UTC m=+7.798117286" watchObservedRunningTime="2026-11-03 17:33:18.653748171 +0000 UTC m=+7.799215768"
Nov 03 17:33:18.909265 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:18.908788    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal" podStartSLOduration=6.908765282 podStartE2EDuration="6.908765282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:18.779544092 +0000 UTC m=+7.925011692" watchObservedRunningTime="2026-11-03 17:33:18.908765282 +0000 UTC m=+8.054232872"
Nov 03 17:33:19.423801 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:19.423763    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"00fb4e1599f1bd3962469eeb7b079336","Type":"ContainerStarted","Data":"322e95c95a2335f1663ebc00932a5cb4c4e7e7593a7eed04030ff8a5dd7f915a"}
Nov 03 17:33:19.429377 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:19.429343    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kpf92" event={"ID":"4b47b5da-225b-4e79-aab7-f81b12aa6191","Type":"ContainerStarted","Data":"3180dfbef5920c2870b48f78d1d11ee83b90b7f75536040554ad44fcc7597296"}
Nov 03 17:33:19.663659 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:19.663474    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kpf92" podStartSLOduration=1.794071548 podStartE2EDuration="7.663457814s" totalImagesPullingTime="5.869386266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:19.661539067 +0000 UTC m=+8.807006655" watchObservedRunningTime="2026-11-03 17:33:19.663457814 +0000 UTC m=+8.808925406"
Nov 03 17:33:19.664186 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:19.664159    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" podStartSLOduration=7.664151747 podStartE2EDuration="7.664151747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:19.543499358 +0000 UTC m=+8.688966951" watchObservedRunningTime="2026-11-03 17:33:19.664151747 +0000 UTC m=+8.809619340"
Nov 03 17:33:20.190618 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.190050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:20.190618 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.190164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:20.190618 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.190326    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:20.190618 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.190399    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:33:28.190378367 +0000 UTC m=+17.335845958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:20.190618 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.190529    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:20.190618 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.190570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:33:28.190557942 +0000 UTC m=+17.336025517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:20.291590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.291511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:20.291761 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.291669    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:20.291761 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.291688    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:20.291761 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.291700    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:20.291925 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.291761    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:33:28.291747105 +0000 UTC m=+17.437214676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:20.331642 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.331602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:20.331851 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.331786    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:20.331974 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.331863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:20.331974 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.331936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:20.332177 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.331985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:20.332177 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:20.332049    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:20.435929 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.435864    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v" event={"ID":"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d","Type":"ContainerStarted","Data":"ba0fd273a29ded31416eb74ac1abfc18a3a7d8379ecee4333c52c3be3c92717c"}
Nov 03 17:33:20.647490 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.647391    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:20.648287 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:20.648261    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:21.444188 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:21.444160    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rxbv7" event={"ID":"7f48d878-d642-49c6-a4db-6e74ea2542fc","Type":"ContainerStarted","Data":"8b7203b698151c9bc7cc5a0f02d5a5d01ae0f7d76bbae0653ed043304080df24"}
Nov 03 17:33:21.506363 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:21.506316    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:33:21.566049 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:21.565985    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rxbv7" podStartSLOduration=2.367476874 podStartE2EDuration="10.565968009s" totalImagesPullingTime="8.198491135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:21.565636887 +0000 UTC m=+10.711104480" watchObservedRunningTime="2026-11-03 17:33:21.565968009 +0000 UTC m=+10.711435645"
Nov 03 17:33:22.289413 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.289312    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:33:21.506334988Z","UUID":"c410059e-2096-402d-962d-c2afae6e3598","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:33:22.291212 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.291187    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:33:22.291331 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.291232    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:33:22.331138 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.331100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:22.331374 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:22.331347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:22.331509 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.331438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:22.331567 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:22.331519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:22.331675 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.331660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:22.331777 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:22.331761    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:22.451033 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.450945    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v" event={"ID":"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d","Type":"ContainerStarted","Data":"e995334da36a84e54755e72002c3b931ac2b9ba29a7057c5cfe25dacfebf6d69"}
Nov 03 17:33:22.469190 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:22.464014    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerStarted","Data":"67698d4652b85b5f7b2ef7d87b93f8cec3f4aacc9074280a17e9d9d649858097"}
Nov 03 17:33:23.468448 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:23.468410    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v" event={"ID":"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d","Type":"ContainerStarted","Data":"b347fc136954485811531c1a0e9bf2388dc138b3642e5789b1f7786d27f8f20b"}
Nov 03 17:33:23.470485 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:23.470454    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="67698d4652b85b5f7b2ef7d87b93f8cec3f4aacc9074280a17e9d9d649858097" exitCode=0
Nov 03 17:33:23.470593 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:23.470542    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerDied","Data":"67698d4652b85b5f7b2ef7d87b93f8cec3f4aacc9074280a17e9d9d649858097"}
Nov 03 17:33:23.472564 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:23.472450    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdjkq" event={"ID":"6544fcbf-d2a3-48fc-8f79-71cc649dd82b","Type":"ContainerStarted","Data":"6175a0d50e613262cf0f76aae4890a533b89d3ef84a0387bb31e96ca2b56f7e3"}
Nov 03 17:33:23.718330 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:23.718210    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v" podStartSLOduration=2.070340377 podStartE2EDuration="11.718188779s" totalImagesPullingTime="9.647848402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:23.593001889 +0000 UTC m=+12.738469481" watchObservedRunningTime="2026-11-03 17:33:23.718188779 +0000 UTC m=+12.863656372"
Nov 03 17:33:23.836923 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:23.836871    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sdjkq" podStartSLOduration=4.354902006 podStartE2EDuration="12.83685688s" totalImagesPullingTime="8.481954874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:23.836782295 +0000 UTC m=+12.982249882" watchObservedRunningTime="2026-11-03 17:33:23.83685688 +0000 UTC m=+12.982324471"
Nov 03 17:33:24.331880 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:24.331843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:24.331880 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:24.331869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:24.331880 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:24.331895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:24.332161 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:24.331995    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:24.332206 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:24.332160    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:24.332306 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:24.332252    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:24.476681 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:24.476634    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xt6l" event={"ID":"1df9489d-1992-48e2-9fca-b6a83d7a0cff","Type":"ContainerStarted","Data":"daec75287860dff930e147d8598240754c878ec335b5d3741fdd7dacf36bdd8f"}
Nov 03 17:33:24.597366 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:24.597266    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7xt6l" podStartSLOduration=2.068412732 podStartE2EDuration="12.597251503s" totalImagesPullingTime="10.528838771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:24.596662663 +0000 UTC m=+13.742130255" watchObservedRunningTime="2026-11-03 17:33:24.597251503 +0000 UTC m=+13.742719095"
Nov 03 17:33:25.481412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:25.481379    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="12fd936c1541d8c9e9aaa9745a9eae0d492bf715fabb1fe95d8c763041f787e2" exitCode=0
Nov 03 17:33:25.482058 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:25.481470    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerDied","Data":"12fd936c1541d8c9e9aaa9745a9eae0d492bf715fabb1fe95d8c763041f787e2"}
Nov 03 17:33:26.330852 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:26.330821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:26.330852 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:26.330849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:26.331053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:26.330926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:26.331053 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:26.330947    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:26.331053 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:26.331005    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:26.331195 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:26.331072    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:27.487200 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.487160    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bs2m4" event={"ID":"dbb54bc0-f60d-4745-bcfe-5978119c0bde","Type":"ContainerStarted","Data":"a1262ccd24d9c5832d1ef81e5189d67ae5dac0fb308faf45f3f09a6d850c48a4"}
Nov 03 17:33:27.489418 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.489394    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="4a39e8e55dfeb0758af2d78cbb8715f8f9bef5f7e8ce2bbb0cc8f10b892db2fe" exitCode=0
Nov 03 17:33:27.489538 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.489434    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerDied","Data":"4a39e8e55dfeb0758af2d78cbb8715f8f9bef5f7e8ce2bbb0cc8f10b892db2fe"}
Nov 03 17:33:27.591627 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.591463    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:27.591745 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.591723    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:27.592189 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.592165    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:33:27.606000 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:27.605954    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bs2m4" podStartSLOduration=1.183235013 podStartE2EDuration="15.605938284s" totalImagesPullingTime="14.422703271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:27.605391359 +0000 UTC m=+16.750858950" watchObservedRunningTime="2026-11-03 17:33:27.605938284 +0000 UTC m=+16.751405876"
Nov 03 17:33:28.258711 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.258622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:28.258711 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.258707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:28.258940 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.258805    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:28.258940 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.258848    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:28.258940 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.258867    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.258853285 +0000 UTC m=+33.404320855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:28.258940 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.258899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.258887524 +0000 UTC m=+33.404355095 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:28.331107 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.331071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:28.331274 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.331136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:28.331274 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.331224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:28.331274 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.331237    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:28.331372 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.331304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:28.331372 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.331366    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:28.360019 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.359987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:28.360176 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.360158    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:28.360216 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.360182    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:28.360216 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.360192    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:28.360298 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:28.360246    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.360232248 +0000 UTC m=+33.505699819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:28.495260 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:33:28.495782 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495613    2579 generic.go:356] "Generic (PLEG): container finished" podID="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" containerID="74b847545ab15ba508effd70e706e9d242671b3f97b9fa3a6e5a87b601a185ce" exitCode=1
Nov 03 17:33:28.495782 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495680    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"dc08879998d34046e25f136935425aef0a1f758b89023a069f608c047731ef2c"}
Nov 03 17:33:28.495900 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495783    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"7f1c62c2db0d38f94f1691510732c45cd297ae7202a774af44b54932540d8d92"}
Nov 03 17:33:28.495900 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495822    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"5fe080a072a5664660a5e0f35c2b9ed5986492df4e5af0d363951c3f0ffbc4b0"}
Nov 03 17:33:28.495998 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495866    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"d3871ccf8613504f1a5bf65d85781e9c3b9a302edbce0a5becf45c68fdd13606"}
Nov 03 17:33:28.496052 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.495999    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerDied","Data":"74b847545ab15ba508effd70e706e9d242671b3f97b9fa3a6e5a87b601a185ce"}
Nov 03 17:33:28.496099 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:28.496066    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"ddc87c16409896254e81d63bbc1b8c32dc757c817a9e340128c57b35c12bdbbf"}
Nov 03 17:33:29.501269 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:29.501077    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerStarted","Data":"9894e6736069653b1ed9583a9c254c2f62e89498bb138dc5cdb22e2cec4b72a2"}
Nov 03 17:33:30.331126 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.331087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:30.331287 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.331153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:30.331287 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.331265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:30.331416 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:30.331280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:30.331416 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:30.331350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:30.331416 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:30.331399    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:30.505940 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.505847    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="9894e6736069653b1ed9583a9c254c2f62e89498bb138dc5cdb22e2cec4b72a2" exitCode=0
Nov 03 17:33:30.505940 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.505906    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerDied","Data":"9894e6736069653b1ed9583a9c254c2f62e89498bb138dc5cdb22e2cec4b72a2"}
Nov 03 17:33:30.509235 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.509219    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:33:30.509580 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:30.509562    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"b9aece4b4e9a7c9b11c7884cd48430420601bca1ff50777233bc30decaa43ab4"}
Nov 03 17:33:31.003689 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:31.003612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:31.226988 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:31.226959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:31.386002 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:31.385917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:31.515160 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:31.515125    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="19a07305db1c2e2061ff5c3dd2d7038a2fcbd171f617946d40e9d453be582db5" exitCode=0
Nov 03 17:33:31.515653 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:31.515178    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerDied","Data":"19a07305db1c2e2061ff5c3dd2d7038a2fcbd171f617946d40e9d453be582db5"}
Nov 03 17:33:31.930515 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:31.930485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:32.090053 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.090020    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:32.331145 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.331108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:32.331298 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.331151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:32.331298 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.331108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:32.331298 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:32.331245    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:32.331413 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:32.331312    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:32.331413 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:32.331386    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:32.521381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.521274    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" event={"ID":"9e659c14-b9ea-4485-a903-eb9ed8e42272","Type":"ContainerStarted","Data":"43ff5df518f56eddaeed3c203b9fcf345267d44955f61657e11da32bc5f8429e"}
Nov 03 17:33:32.524572 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.524548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:33:32.524929 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.524904    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"c539e549d3779a9e845baa2b3927b479dd7ffa9e009daccf3844f864641813ca"}
Nov 03 17:33:32.525243 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.525214    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:32.525367 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.525354    2579 scope.go:122] "RemoveContainer" containerID="74b847545ab15ba508effd70e706e9d242671b3f97b9fa3a6e5a87b601a185ce"
Nov 03 17:33:32.542922 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.542749    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:32.647401 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:32.647341    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4tnrc" podStartSLOduration=9.561799447 podStartE2EDuration="21.647326988s" totalImagesPullingTime="12.085527541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:32.646703548 +0000 UTC m=+21.792171140" watchObservedRunningTime="2026-11-03 17:33:32.647326988 +0000 UTC m=+21.792794579"
Nov 03 17:33:33.535400 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.535363    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:33:33.536412 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.536362    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" event={"ID":"5908cbf5-2e94-4a1c-9c81-f7adfd472da0","Type":"ContainerStarted","Data":"1367372e4f5019c06abdd688554eb05d6e4c6aebcdf39bac4effe206545a0989"}
Nov 03 17:33:33.536634 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.536603    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:33.537161 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.537139    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:33.552799 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.552773    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:33.593119 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.593091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:33.757056 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.757021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:33.821596 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:33.821500    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd" podStartSLOduration=8.11861904 podStartE2EDuration="22.821484809s" totalImagesPullingTime="14.702865769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:33.771336548 +0000 UTC m=+22.916804141" watchObservedRunningTime="2026-11-03 17:33:33.821484809 +0000 UTC m=+22.966952401"
Nov 03 17:33:34.331242 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:34.331212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:34.331400 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:34.331272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:34.331400 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:34.331355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:34.331400 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:34.331364    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:34.331499 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:34.331417    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:34.331499 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:34.331479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:34.538810 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:34.538785    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:35.090304 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.090228    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cz52j"]
Nov 03 17:33:35.090471 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.090384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:35.090530 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:35.090466    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:35.093231 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.093197    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvv9c"]
Nov 03 17:33:35.093392 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.093334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:35.093461 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:35.093441    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:35.093888 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.093870    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8vqsh"]
Nov 03 17:33:35.093996 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.093981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:35.094203 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:35.094091    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:35.541424 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:35.541355    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:36.331550 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:36.331524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:36.331661 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:36.331639    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:36.517099 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:36.517045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:36.680726 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:36.680651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:37.181621 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:37.181574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:37.331545 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:37.331515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:37.331712 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:37.331623    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:37.331712 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:37.331653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:37.331712 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:37.331703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:37.343231 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:37.343206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:37.515175 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:37.515058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:37.685263 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:37.685193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:38.331749 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:38.331704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:38.331927 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:38.331865    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:38.786209 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:38.786174    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:38.786627 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:38.786377    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:38.798339 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:38.798310    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:33:38.847398 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:38.847350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:39.015436 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:39.015394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:39.297054 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:39.297017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:39.331290 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:39.331257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:39.331290 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:39.331303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:39.331479 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:39.331417    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:39.331546 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:39.331522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:40.331231 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:40.331211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:40.331506 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:40.331328    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:41.030335 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:41.030278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:41.332149 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:41.332045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:41.332590 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:41.332164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:41.332590 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:41.332212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:41.332590 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:41.332347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:42.331716 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:42.331678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:42.331925 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:42.331812    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:43.331067 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:43.331037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:43.331067 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:43.331072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:43.331500 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:43.331142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:43.331500 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:43.331221    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:44.274836 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:44.274793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:44.275035 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:44.274864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:44.275035 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.274948    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:44.275035 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.274947    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:44.275035 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.275001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret podName:46723e8e-0ca8-4209-9904-b35885a2f90f nodeName:}" failed. No retries permitted until 2026-11-03 17:34:16.274987166 +0000 UTC m=+65.420454749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret") pod "global-pull-secret-syncer-cz52j" (UID: "46723e8e-0ca8-4209-9904-b35885a2f90f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:44.275035 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.275014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs podName:634768ef-d8f7-474d-9ee4-b479a1b1f17b nodeName:}" failed. No retries permitted until 2026-11-03 17:34:16.275008408 +0000 UTC m=+65.420475982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs") pod "network-metrics-daemon-8vqsh" (UID: "634768ef-d8f7-474d-9ee4-b479a1b1f17b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:44.331104 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:44.331067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:44.331466 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.331219    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:44.375543 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:44.375502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:44.375715 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.375642    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:44.375715 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.375659    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:44.375715 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.375668    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zkzb4 for pod openshift-network-diagnostics/network-check-target-tvv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:44.375854 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:44.375722    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4 podName:a2260c22-12fe-411e-9a84-e8cd21d7389d nodeName:}" failed. No retries permitted until 2026-11-03 17:34:16.37570658 +0000 UTC m=+65.521174157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkzb4" (UniqueName: "kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4") pod "network-check-target-tvv9c" (UID: "a2260c22-12fe-411e-9a84-e8cd21d7389d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:45.331077 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:45.331040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:45.331255 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:45.331152    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:45.331255 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:45.331204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:45.331584 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:45.331304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:46.331082 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:46.331042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:46.331254 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:46.331159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:47.331353 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:47.331312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:47.331353 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:47.331341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:47.331810 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:47.331429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:47.331810 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:47.331562    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:48.331577 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:48.331546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:48.331980 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:48.331658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:48.367063 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:48.367033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bs2m4_dbb54bc0-f60d-4745-bcfe-5978119c0bde/dns-node-resolver/0.log"
Nov 03 17:33:49.331303 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:49.331270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:49.331303 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:49.331306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:49.331499 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:49.331371    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:49.331499 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:49.331487    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:49.755915 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:49.755890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kpf92_4b47b5da-225b-4e79-aab7-f81b12aa6191/node-ca/0.log"
Nov 03 17:33:50.331002 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:50.330967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:50.331181 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:50.331083    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:51.331936 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:51.331898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:51.332353 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:51.332030    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:51.332353 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:51.332087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:51.332353 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:51.332157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:52.094166 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:52.094127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:52.308559 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:52.308526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:52.331483 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:52.331462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:52.331584 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:52.331568    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:52.504454 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:52.504413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:52.700038 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:52.700006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:52.883405 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:52.883322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:53.330852 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:53.330825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:53.331026 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:53.330830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:53.331026 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:53.330933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:53.331026 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:53.330993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:53.931971 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:53.931938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:54.091769 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.091696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:54.253875 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.253784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:54.331141 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.331109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:54.331320 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:54.331219    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:54.412582 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.412516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:54.578273 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.578197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:54.734234 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.734181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:54.901621 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:54.901591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:55.248485 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:55.248382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:55.330757 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:55.330714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:55.330757 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:55.330725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:55.330970 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:55.330844    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:55.330970 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:55.330942    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:56.086095 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.086064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:56.246260 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.246226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:56.330749 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.330707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:56.331126 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:56.330848    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:56.405250 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.405216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:56.565804 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.565772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:56.723328 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.723227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:56.884661 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:56.884633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:57.041904 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:57.041813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:57.286264 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:57.286233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:57.331679 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:57.331622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:57.331679 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:57.331655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:57.332134 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:57.331747    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:57.332134 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:57.331807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:33:58.331250 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:58.331212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:33:58.331430 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:58.331330    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:33:58.823914 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:58.823886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:58.983108 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:58.983044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:59.331634 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:59.331597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:33:59.331844 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:59.331714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:33:59.331844 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:33:59.331805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:33:59.331968 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:33:59.331921    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:34:00.331275 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:00.331227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:00.331673 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:34:00.331353    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vqsh" podUID="634768ef-d8f7-474d-9ee4-b479a1b1f17b"
Nov 03 17:34:01.332450 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:01.332410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:01.332904 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:01.332427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:01.332904 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:34:01.332521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvv9c" podUID="a2260c22-12fe-411e-9a84-e8cd21d7389d"
Nov 03 17:34:01.332904 ip-10-0-139-156 kubenswrapper[2579]: E1103 17:34:01.332590    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz52j" podUID="46723e8e-0ca8-4209-9904-b35885a2f90f"
Nov 03 17:34:01.476238 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:01.476153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:01.653919 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:01.653887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:01.817140 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:01.817105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:01.972398 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:01.972294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:02.131264 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.131239    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeReady"
Nov 03 17:34:02.131419 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.131365    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 17:34:02.133323 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.133237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:02.287167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.287083    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n7qdx"]
Nov 03 17:34:02.288911 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.288894    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lnx9n"]
Nov 03 17:34:02.289024 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.289009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.290438 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.290416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.306901 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.306876    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n7qdx"]
Nov 03 17:34:02.309793 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.309768    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lnx9n"]
Nov 03 17:34:02.331285 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.331255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:02.346314 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.346287    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rtvjp"]
Nov 03 17:34:02.346652 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.346490    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:34:02.346712 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.346697    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:34:02.348055 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.348038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.348276 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.348246    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:34:02.348420 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.348396    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:34:02.348420 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.348409    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstfb\""
Nov 03 17:34:02.348554 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.348454    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:34:02.348591 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.348566    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:34:02.401946 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.401908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07057c45-a937-446c-96ea-c3e74244776b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.401946 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.401944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0481c4-d60c-47f9-9fc5-901a405928bc-cert\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.402165 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.401964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62slh\" (UniqueName: \"kubernetes.io/projected/07057c45-a937-446c-96ea-c3e74244776b-kube-api-access-62slh\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.402165 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.402079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprfg\" (UniqueName: \"kubernetes.io/projected/ea0481c4-d60c-47f9-9fc5-901a405928bc-kube-api-access-wprfg\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.402165 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.402152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07057c45-a937-446c-96ea-c3e74244776b-data-volume\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.402273 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.402197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07057c45-a937-446c-96ea-c3e74244776b-crio-socket\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.402273 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.402224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07057c45-a937-446c-96ea-c3e74244776b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.411201 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.411177    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rtvjp"]
Nov 03 17:34:02.448688 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.448652    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:34:02.448875 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.448724    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:34:02.448875 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.448871    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pwfx\""
Nov 03 17:34:02.449041 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.449030    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:34:02.449115 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.449095    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:34:02.449158 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.449113    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\""
Nov 03 17:34:02.449192 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.449176    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jj5j\""
Nov 03 17:34:02.502987 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.502953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprfg\" (UniqueName: \"kubernetes.io/projected/ea0481c4-d60c-47f9-9fc5-901a405928bc-kube-api-access-wprfg\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.503167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07057c45-a937-446c-96ea-c3e74244776b-data-volume\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07057c45-a937-446c-96ea-c3e74244776b-crio-socket\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07057c45-a937-446c-96ea-c3e74244776b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-tmp-dir\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.503167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-metrics-tls\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.503404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tph\" (UniqueName: \"kubernetes.io/projected/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-kube-api-access-59tph\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.503404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07057c45-a937-446c-96ea-c3e74244776b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0481c4-d60c-47f9-9fc5-901a405928bc-cert\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.503404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-config-volume\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.503404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62slh\" (UniqueName: \"kubernetes.io/projected/07057c45-a937-446c-96ea-c3e74244776b-kube-api-access-62slh\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07057c45-a937-446c-96ea-c3e74244776b-crio-socket\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503677 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07057c45-a937-446c-96ea-c3e74244776b-data-volume\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.503963 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.503944    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07057c45-a937-446c-96ea-c3e74244776b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.506835 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.506812    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07057c45-a937-446c-96ea-c3e74244776b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.506930 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.506913    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0481c4-d60c-47f9-9fc5-901a405928bc-cert\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.568165 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.568098    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62slh\" (UniqueName: \"kubernetes.io/projected/07057c45-a937-446c-96ea-c3e74244776b-kube-api-access-62slh\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.568334 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.568307    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprfg\" (UniqueName: \"kubernetes.io/projected/ea0481c4-d60c-47f9-9fc5-901a405928bc-kube-api-access-wprfg\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.602848 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.602816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:02.604000 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.603978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-config-volume\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.604055 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.604044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-tmp-dir\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.604088 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.604074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-metrics-tls\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.604124 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.604088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tph\" (UniqueName: \"kubernetes.io/projected/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-kube-api-access-59tph\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.604388 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.604363    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-tmp-dir\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.604592 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.604565    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-config-volume\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.604811 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.604792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:02.606583 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.606564    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-metrics-tls\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.643517 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.643463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:02.673547 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.673513    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tph\" (UniqueName: \"kubernetes.io/projected/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-kube-api-access-59tph\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.726285 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.726254    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"]
Nov 03 17:34:02.728591 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.728573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:02.783289 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.783246    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v8zr6\""
Nov 03 17:34:02.783456 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.783359    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 17:34:02.788375 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.788351    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"]
Nov 03 17:34:02.804845 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.804801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5739357-c9b8-445d-9b36-137395fd0857-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tb5x7\" (UID: \"c5739357-c9b8-445d-9b36-137395fd0857\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:02.806444 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.806417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:02.906261 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.906227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5739357-c9b8-445d-9b36-137395fd0857-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tb5x7\" (UID: \"c5739357-c9b8-445d-9b36-137395fd0857\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:02.908667 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.908647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5739357-c9b8-445d-9b36-137395fd0857-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tb5x7\" (UID: \"c5739357-c9b8-445d-9b36-137395fd0857\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:02.960866 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.960825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:02.989230 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:02.989199    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lnx9n"]
Nov 03 17:34:02.993598 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:02.993573    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0481c4_d60c_47f9_9fc5_901a405928bc.slice/crio-336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f WatchSource:0}: Error finding container 336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f: Status 404 returned error can't find the container with id 336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f
Nov 03 17:34:03.038054 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.038020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:03.040877 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.040848    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n7qdx"]
Nov 03 17:34:03.045310 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:03.045285    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07057c45_a937_446c_96ea_c3e74244776b.slice/crio-49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2 WatchSource:0}: Error finding container 49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2: Status 404 returned error can't find the container with id 49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2
Nov 03 17:34:03.334063 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.333991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:03.334063 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.334023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:03.336536 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.336512    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rtvjp"]
Nov 03 17:34:03.340300 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:03.340259    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb9dbee_5655_45fc_8d5c_a51cfa17ea7d.slice/crio-c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360 WatchSource:0}: Error finding container c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360: Status 404 returned error can't find the container with id c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360
Nov 03 17:34:03.388469 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.388424    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:34:03.388469 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.388424    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:34:03.388999 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.388935    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\""
Nov 03 17:34:03.388999 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.388987    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:34:03.389063 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.388987    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\""
Nov 03 17:34:03.409645 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.409615    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"]
Nov 03 17:34:03.412868 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:03.412846    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5739357_c9b8_445d_9b36_137395fd0857.slice/crio-76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40 WatchSource:0}: Error finding container 76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40: Status 404 returned error can't find the container with id 76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40
Nov 03 17:34:03.602963 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.602874    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lnx9n" event={"ID":"ea0481c4-d60c-47f9-9fc5-901a405928bc","Type":"ContainerStarted","Data":"336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f"}
Nov 03 17:34:03.603978 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.603956    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7" event={"ID":"c5739357-c9b8-445d-9b36-137395fd0857","Type":"ContainerStarted","Data":"76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40"}
Nov 03 17:34:03.604967 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.604943    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rtvjp" event={"ID":"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d","Type":"ContainerStarted","Data":"c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360"}
Nov 03 17:34:03.606269 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.606249    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7qdx" event={"ID":"07057c45-a937-446c-96ea-c3e74244776b","Type":"ContainerStarted","Data":"9cf1a537c4717a7b072531c54ca56d76bab43ddaa15fc757b221df379e30fd70"}
Nov 03 17:34:03.606356 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:03.606287    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7qdx" event={"ID":"07057c45-a937-446c-96ea-c3e74244776b","Type":"ContainerStarted","Data":"49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2"}
Nov 03 17:34:04.611579 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:04.611486    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7qdx" event={"ID":"07057c45-a937-446c-96ea-c3e74244776b","Type":"ContainerStarted","Data":"7fa0c356209cc79616d33360419d5521abdd1abedf472a5d7aed8233457c4611"}
Nov 03 17:34:05.617760 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:05.617253    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7" event={"ID":"c5739357-c9b8-445d-9b36-137395fd0857","Type":"ContainerStarted","Data":"e3aa27dad2f9dd47c83fac566b3fdbfd5f59a6733d1a134c4d3e951a99e82911"}
Nov 03 17:34:05.617760 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:05.617678    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:05.623622 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:05.623569    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:05.757157 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:05.756981    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7" podStartSLOduration=2.176933961 podStartE2EDuration="3.756965057s" totalImagesPullingTime="1.580031096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:05.755847145 +0000 UTC m=+54.901314737" watchObservedRunningTime="2026-11-03 17:34:05.756965057 +0000 UTC m=+54.902432649"
Nov 03 17:34:06.103794 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.103757    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f7fc76775-qdhqf"]
Nov 03 17:34:06.105858 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.105835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.160691 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160663    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 17:34:06.160890 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160700    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 17:34:06.160890 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160663    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 17:34:06.160890 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160840    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 17:34:06.161061 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160900    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 17:34:06.161061 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160910    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 17:34:06.161061 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160933    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 17:34:06.161061 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.160946    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6mmfj\""
Nov 03 17:34:06.210075 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.210046    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7fc76775-qdhqf"]
Nov 03 17:34:06.237883 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.237856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.237883 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.237888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.238098 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.237904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.238098 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.238000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.238098 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.238052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.238098 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.238071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.238267 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.238107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.338802 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.338772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.338945 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.338819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.338945 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.338934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339017 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.338965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339167 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339229 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339277 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339607 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339520    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339857 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339836    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339933 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339869    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.339933 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.339923    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.341468 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.341441    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.341468 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.341463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.402607 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.402568    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.418936 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.418868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:06.621659 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.621627    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rtvjp" event={"ID":"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d","Type":"ContainerStarted","Data":"530bfbcc38f9b47ce91c17f5a88d1a3d1a63325028a6c2911b8bd13ce3e85ea4"}
Nov 03 17:34:06.622127 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.621687    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rtvjp" event={"ID":"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d","Type":"ContainerStarted","Data":"7e5d3a6ff094227f07318f388038762fdfdfcfd21e8acb24eec5a1735957aeb4"}
Nov 03 17:34:06.622127 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.621720    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:06.623777 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.623725    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7qdx" event={"ID":"07057c45-a937-446c-96ea-c3e74244776b","Type":"ContainerStarted","Data":"6ada520a7047240671cf0a5d44664b46424590b96e12004edeb7928c3225e7ab"}
Nov 03 17:34:06.742766 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.742643    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rtvjp" podStartSLOduration=2.303815926 podStartE2EDuration="4.742627722s" totalImagesPullingTime="2.438811796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:06.742304429 +0000 UTC m=+55.887772021" watchObservedRunningTime="2026-11-03 17:34:06.742627722 +0000 UTC m=+55.888095311"
Nov 03 17:34:06.859237 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.859198    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7fc76775-qdhqf"]
Nov 03 17:34:06.862960 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:06.862905    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3044c8b1_0ded_48b6_8496_5efad747ee0a.slice/crio-4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2 WatchSource:0}: Error finding container 4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2: Status 404 returned error can't find the container with id 4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2
Nov 03 17:34:06.863600 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.863549    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n7qdx" podStartSLOduration=1.789563666 podStartE2EDuration="4.863535298s" totalImagesPullingTime="3.073971632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:34:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:06.862980145 +0000 UTC m=+56.008447734" watchObservedRunningTime="2026-11-03 17:34:06.863535298 +0000 UTC m=+56.009002889"
Nov 03 17:34:06.929915 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:06.929880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:07.087948 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.087877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:07.240623 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.240595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:07.347611 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.347533    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"]
Nov 03 17:34:07.349905 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.349888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.403302 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.403265    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 17:34:07.403459 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.403358    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b97jd\""
Nov 03 17:34:07.403459 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.403414    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 17:34:07.404330 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.404314    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:34:07.404381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.404334    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:34:07.404381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.404314    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:34:07.449489 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.449454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfjm\" (UniqueName: \"kubernetes.io/projected/c4d0a499-8879-4147-b92b-b81d727ddff3-kube-api-access-jwfjm\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.449489 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.449492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.449764 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.449520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a499-8879-4147-b92b-b81d727ddff3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.449764 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.449645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.453879 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.453854    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"]
Nov 03 17:34:07.505910 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.505868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:07.550984 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.550930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfjm\" (UniqueName: \"kubernetes.io/projected/c4d0a499-8879-4147-b92b-b81d727ddff3-kube-api-access-jwfjm\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.550984 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.550986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.551234 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.551109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a499-8879-4147-b92b-b81d727ddff3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.551234 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.551154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.551898 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.551876    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a499-8879-4147-b92b-b81d727ddff3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.554006 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.553983    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.554119 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.554095    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.615086 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.615017    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfjm\" (UniqueName: \"kubernetes.io/projected/c4d0a499-8879-4147-b92b-b81d727ddff3-kube-api-access-jwfjm\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:07.628456 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.628420    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7fc76775-qdhqf" event={"ID":"3044c8b1-0ded-48b6-8496-5efad747ee0a","Type":"ContainerStarted","Data":"4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2"}
Nov 03 17:34:07.660696 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:07.660656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:08.040768 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:08.040715    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"]
Nov 03 17:34:08.044988 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:08.044951    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d0a499_8879_4147_b92b_b81d727ddff3.slice/crio-335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb WatchSource:0}: Error finding container 335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb: Status 404 returned error can't find the container with id 335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb
Nov 03 17:34:08.634047 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:08.634002    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lnx9n" event={"ID":"ea0481c4-d60c-47f9-9fc5-901a405928bc","Type":"ContainerStarted","Data":"64c815f718d3394a257907d0a103d7f7f7d34ae977c533ac53366ffd7514aa6a"}
Nov 03 17:34:08.635615 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:08.635578    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw" event={"ID":"c4d0a499-8879-4147-b92b-b81d727ddff3","Type":"ContainerStarted","Data":"335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb"}
Nov 03 17:34:08.753182 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:08.753071    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lnx9n" podStartSLOduration=1.350729047 podStartE2EDuration="6.753050111s" totalImagesPullingTime="5.402321064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:08.752583395 +0000 UTC m=+57.898050988" watchObservedRunningTime="2026-11-03 17:34:08.753050111 +0000 UTC m=+57.898517707"
Nov 03 17:34:10.647931 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:10.647895    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw" event={"ID":"c4d0a499-8879-4147-b92b-b81d727ddff3","Type":"ContainerStarted","Data":"f17f1e2f82197c17fd19408ea0982ba5d0c361d7f0ef58cbc11fe66e0ad53728"}
Nov 03 17:34:10.648413 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:10.647989    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw" event={"ID":"c4d0a499-8879-4147-b92b-b81d727ddff3","Type":"ContainerStarted","Data":"bd47fd3f73e5775f0e2a413184213ff8336a4a0c9d7f9a1c3ca920f98ca1ae66"}
Nov 03 17:34:10.649525 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:10.649502    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7fc76775-qdhqf" event={"ID":"3044c8b1-0ded-48b6-8496-5efad747ee0a","Type":"ContainerStarted","Data":"144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af"}
Nov 03 17:34:10.768835 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:10.768777    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw" podStartSLOduration=2.121112944 podStartE2EDuration="3.768763823s" totalImagesPullingTime="1.647650879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:10.76795481 +0000 UTC m=+59.913422402" watchObservedRunningTime="2026-11-03 17:34:10.768763823 +0000 UTC m=+59.914231415"
Nov 03 17:34:10.888946 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:10.888885    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f7fc76775-qdhqf" podStartSLOduration=1.656047496 podStartE2EDuration="4.888866022s" totalImagesPullingTime="3.232818526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:10.888271264 +0000 UTC m=+60.033738892" watchObservedRunningTime="2026-11-03 17:34:10.888866022 +0000 UTC m=+60.034333614"
Nov 03 17:34:12.382887 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:12.382854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:13.264471 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:13.264434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:13.468108 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:13.468078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:13.790003 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:13.789964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:13.951532 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:13.951497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:13.992947 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:13.992916    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"]
Nov 03 17:34:13.995582 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:13.995560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.048856 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.048771    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nctqr\""
Nov 03 17:34:14.050253 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.050230    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 17:34:14.050253 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.050234    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:34:14.057018 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.056989    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"]
Nov 03 17:34:14.103046 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.103007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.103223 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.103058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.103223 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.103184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.103223 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.103213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6tv\" (UniqueName: \"kubernetes.io/projected/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-kube-api-access-xv6tv\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.105457 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.105428    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"]
Nov 03 17:34:14.107804 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.107782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.108931 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.108907    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9znfw"]
Nov 03 17:34:14.110781 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.110763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.122925 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.122901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:14.162532 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.162491    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:34:14.162532 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.162492    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 17:34:14.162532 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.162525    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 17:34:14.162793 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.162552    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mvxtn\""
Nov 03 17:34:14.163801 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.163780    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:34:14.164042 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.164027    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:34:14.164102 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.164091    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:34:14.164171 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.164162    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8sjq\""
Nov 03 17:34:14.173820 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.173793    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"]
Nov 03 17:34:14.203855 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.203818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-sys\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.203855 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.203860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.204085 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.203881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-wtmp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204085 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.203902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c628272-7a16-4897-af12-59bcfedd74cb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.204085 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-root\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204085 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwrp\" (UniqueName: \"kubernetes.io/projected/cb2852e1-7207-4d27-a78e-ab6a06a425c3-kube-api-access-snwrp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204258 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.204258 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.204258 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-metrics-client-ca\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204407 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-tls\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204407 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6tv\" (UniqueName: \"kubernetes.io/projected/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-kube-api-access-xv6tv\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.204407 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204554 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.204554 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-textfile\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204554 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.204554 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.204776 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.204776 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdz2\" (UniqueName: \"kubernetes.io/projected/1c628272-7a16-4897-af12-59bcfedd74cb-kube-api-access-mgdz2\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.204776 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.204977 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.204957    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.206656 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.206628    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.207030 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.207011    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.267804 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.267764    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6tv\" (UniqueName: \"kubernetes.io/projected/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-kube-api-access-xv6tv\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.290208 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.290172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:14.305374 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-tls\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.305374 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.305374 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-textfile\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdz2\" (UniqueName: \"kubernetes.io/projected/1c628272-7a16-4897-af12-59bcfedd74cb-kube-api-access-mgdz2\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-sys\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-wtmp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c628272-7a16-4897-af12-59bcfedd74cb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.305629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-root\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwrp\" (UniqueName: \"kubernetes.io/projected/cb2852e1-7207-4d27-a78e-ab6a06a425c3-kube-api-access-snwrp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-metrics-client-ca\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305832    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-textfile\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.305940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-sys\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306015    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-wtmp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306026    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306136 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306136    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-root\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306697 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306479    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c628272-7a16-4897-af12-59bcfedd74cb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.306697 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306508    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.306697 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306543    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-metrics-client-ca\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.306876 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.306776    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.307056 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.307036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:14.308202 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.308179    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-tls\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.308338 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.308312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.308410 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.308388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.308758 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.308716    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.369247 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.369185    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwrp\" (UniqueName: \"kubernetes.io/projected/cb2852e1-7207-4d27-a78e-ab6a06a425c3-kube-api-access-snwrp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.369247 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.369199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdz2\" (UniqueName: \"kubernetes.io/projected/1c628272-7a16-4897-af12-59bcfedd74cb-kube-api-access-mgdz2\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.421046 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.421006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:14.426487 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.426452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:14.435042 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:14.435017    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2852e1_7207_4d27_a78e_ab6a06a425c3.slice/crio-b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8 WatchSource:0}: Error finding container b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8: Status 404 returned error can't find the container with id b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8
Nov 03 17:34:14.457347 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.457313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:14.624826 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.624724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:14.666176 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.666139    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9znfw" event={"ID":"cb2852e1-7207-4d27-a78e-ab6a06a425c3","Type":"ContainerStarted","Data":"b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8"}
Nov 03 17:34:14.734148 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.734116    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"]
Nov 03 17:34:14.736853 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:14.736820    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3422e3d_3f9a_497d_9eba_50eaa3f7dfc6.slice/crio-751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76 WatchSource:0}: Error finding container 751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76: Status 404 returned error can't find the container with id 751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76
Nov 03 17:34:14.786404 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.786378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:14.794270 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:14.794248    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"]
Nov 03 17:34:14.798321 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:14.798287    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c628272_7a16_4897_af12_59bcfedd74cb.slice/crio-203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a WatchSource:0}: Error finding container 203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a: Status 404 returned error can't find the container with id 203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a
Nov 03 17:34:15.561594 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:15.561561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:15.672896 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:15.672859    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q" event={"ID":"1c628272-7a16-4897-af12-59bcfedd74cb","Type":"ContainerStarted","Data":"203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a"}
Nov 03 17:34:15.675120 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:15.675092    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr" event={"ID":"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6","Type":"ContainerStarted","Data":"92ad7dd5f5da6ad88e40f28ea3f87f17fdc538877cb25d428243b3f0fea802c6"}
Nov 03 17:34:15.675237 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:15.675153    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr" event={"ID":"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6","Type":"ContainerStarted","Data":"ac24accfbf76335f82876ca8f32b578d04d853293898dd55f6ec80cbc19201ea"}
Nov 03 17:34:15.675237 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:15.675180    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr" event={"ID":"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6","Type":"ContainerStarted","Data":"751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76"}
Nov 03 17:34:15.948536 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:15.948503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:16.321953 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.321918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:16.322084 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.322014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:16.341464 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.341437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:16.373522 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.373460    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:34:16.373522 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.373463    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:34:16.385242 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.385214    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:16.385242 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.385232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:16.393125 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.393095    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:34:16.398210 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.398186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.419314 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.419279    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:16.419616 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.419589    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:16.422458 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.422435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:16.424710 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.424688    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:16.453189 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.453162    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 17:34:16.453189 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.453181    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 17:34:16.453411 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.453161    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 17:34:16.453411 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.453210    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 17:34:16.453411 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.453281    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 17:34:16.453411 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.453292    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b2s8h\""
Nov 03 17:34:16.474273 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.474248    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:34:16.523096 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523257 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523257 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523257 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523422 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523557 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523602 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523602 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523704 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523773 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523860 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523901 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.523935 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.523896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.527843 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.527821    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\""
Nov 03 17:34:16.527843 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.527821    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 17:34:16.532889 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.532836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:16.551634 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.551603    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:34:16.552970 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.552948    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:34:16.553435 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.553288    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 17:34:16.553435 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.553302    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 17:34:16.553435 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.553334    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 17:34:16.556054 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.556025    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:16.602521 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.602469    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\""
Nov 03 17:34:16.611435 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.611413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:16.624898 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.624873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625016 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.624908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625016 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.624931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625016 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625155 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625155 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625155 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625416 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625482 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625432    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625482 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.625575 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.625486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.628774 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.628599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.629691 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.629021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.629691 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.629085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.629691 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.629122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.637328 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.631517    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.637328 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.632681    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.637328 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.633663    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.637328 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.634963    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:16.641027 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.640953    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.642669 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.641893    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.642669 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.642078    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.642669 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.642215    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.642669 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.642319    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.642669 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.642629    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.643176 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.643141    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.684069 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.684024    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb2852e1-7207-4d27-a78e-ab6a06a425c3" containerID="9ebf5a77f7d8a0daae06d4f55be5a85028f8d58837a8f8f57d46d325e949c10a" exitCode=0
Nov 03 17:34:16.684528 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.684103    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9znfw" event={"ID":"cb2852e1-7207-4d27-a78e-ab6a06a425c3","Type":"ContainerDied","Data":"9ebf5a77f7d8a0daae06d4f55be5a85028f8d58837a8f8f57d46d325e949c10a"}
Nov 03 17:34:16.692267 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.692146    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q" event={"ID":"1c628272-7a16-4897-af12-59bcfedd74cb","Type":"ContainerStarted","Data":"75436761c219441981b629007180f85be5f794e227a8ad64c27de824b82b3672"}
Nov 03 17:34:16.692267 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.692210    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q" event={"ID":"1c628272-7a16-4897-af12-59bcfedd74cb","Type":"ContainerStarted","Data":"d84a7fbfa041c1f07d2160935d8107a34a3ce3ac5f786fe16cf59489804d19ff"}
Nov 03 17:34:16.697195 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.697149    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.700989 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.700896    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr" event={"ID":"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6","Type":"ContainerStarted","Data":"182a4a9662cc9d5f79e753e32a197269d859c9b7cc3c2a44c64b3080c34de526"}
Nov 03 17:34:16.706845 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.706816    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:16.716504 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.716471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:16.743630 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.743597    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"]
Nov 03 17:34:16.748234 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.748211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.803297 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.803268    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 17:34:16.805567 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.805550    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 17:34:16.805775 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.805756    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 17:34:16.805775 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.805679    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-58h0jjau7t32b\""
Nov 03 17:34:16.805877 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.805648    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 17:34:16.806114 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.806098    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7lr2n\""
Nov 03 17:34:16.812260 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.812236    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"]
Nov 03 17:34:16.898149 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.898118    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\""
Nov 03 17:34:16.906296 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.906273    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 17:34:16.907347 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.907326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:16.915511 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.915485    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8vqsh"]
Nov 03 17:34:16.918915 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:16.918890    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634768ef_d8f7_474d_9ee4_b479a1b1f17b.slice/crio-10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60 WatchSource:0}: Error finding container 10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60: Status 404 returned error can't find the container with id 10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60
Nov 03 17:34:16.930252 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.930225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:16.932396 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932527 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a774fe-c887-4a8c-96a6-ed3b863e344c-metrics-client-ca\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932527 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932527 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932679 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a774fe-c887-4a8c-96a6-ed3b863e344c-tmp\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932679 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932679 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-grpc-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932836 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzwx\" (UniqueName: \"kubernetes.io/projected/25a774fe-c887-4a8c-96a6-ed3b863e344c-kube-api-access-kqzwx\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.932836 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.932807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:16.993323 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:16.993290    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cz52j"]
Nov 03 17:34:16.996920 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:16.996891    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46723e8e_0ca8_4209_9904_b35885a2f90f.slice/crio-f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1 WatchSource:0}: Error finding container f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1: Status 404 returned error can't find the container with id f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1
Nov 03 17:34:17.034029 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.033992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.034208 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.034042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.034208 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.034112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a774fe-c887-4a8c-96a6-ed3b863e344c-tmp\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.034208 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.034146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.034208 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.034167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-grpc-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.034420 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.034211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzwx\" (UniqueName: \"kubernetes.io/projected/25a774fe-c887-4a8c-96a6-ed3b863e344c-kube-api-access-kqzwx\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.034420 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.034245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.035226 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.035194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.035336 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.035262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a774fe-c887-4a8c-96a6-ed3b863e344c-metrics-client-ca\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.035336 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.035279    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a774fe-c887-4a8c-96a6-ed3b863e344c-tmp\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038493 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.035994    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a774fe-c887-4a8c-96a6-ed3b863e344c-metrics-client-ca\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038493 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.037518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038493 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.038143    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038493 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.038432    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-grpc-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038758 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.038600    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038881 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.038859    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.038933 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.038905    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.050491 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.050407    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr" podStartSLOduration=2.612868916 podStartE2EDuration="4.050393288s" totalImagesPullingTime="1.437524372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:17.049388605 +0000 UTC m=+66.194856198" watchObservedRunningTime="2026-11-03 17:34:17.050393288 +0000 UTC m=+66.195860880"
Nov 03 17:34:17.100359 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.100326    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzwx\" (UniqueName: \"kubernetes.io/projected/25a774fe-c887-4a8c-96a6-ed3b863e344c-kube-api-access-kqzwx\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.103394 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.103366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:17.159505 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.159186    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:34:17.165041 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:17.164998    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c4adf8_f640_4c4e_b92b_2c5a7f901b11.slice/crio-9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d WatchSource:0}: Error finding container 9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d: Status 404 returned error can't find the container with id 9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d
Nov 03 17:34:17.288444 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.288417    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvv9c"]
Nov 03 17:34:17.290945 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:17.290913    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2260c22_12fe_411e_9a84_e8cd21d7389d.slice/crio-749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e WatchSource:0}: Error finding container 749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e: Status 404 returned error can't find the container with id 749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e
Nov 03 17:34:17.370993 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.370902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:17.710442 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.710400    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9znfw" event={"ID":"cb2852e1-7207-4d27-a78e-ab6a06a425c3","Type":"ContainerStarted","Data":"350e55759426d2cb9bd448d4b8add9e4d53278c8441456aa21c7fb9c2bb3d706"}
Nov 03 17:34:17.710932 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.710506    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9znfw" event={"ID":"cb2852e1-7207-4d27-a78e-ab6a06a425c3","Type":"ContainerStarted","Data":"f869d34523caaf64c12c7eda85648e9b5f99ca8d835a39420a6f1e4136ad6c3a"}
Nov 03 17:34:17.712598 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.712570    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q" event={"ID":"1c628272-7a16-4897-af12-59bcfedd74cb","Type":"ContainerStarted","Data":"e63c9da52dd2437e392ee58c85b6b357bec270843dd5c428b6d742423735b117"}
Nov 03 17:34:17.713834 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.713810    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvv9c" event={"ID":"a2260c22-12fe-411e-9a84-e8cd21d7389d","Type":"ContainerStarted","Data":"749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e"}
Nov 03 17:34:17.715014 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.714960    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cz52j" event={"ID":"46723e8e-0ca8-4209-9904-b35885a2f90f","Type":"ContainerStarted","Data":"f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1"}
Nov 03 17:34:17.716241 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.716213    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vqsh" event={"ID":"634768ef-d8f7-474d-9ee4-b479a1b1f17b","Type":"ContainerStarted","Data":"10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60"}
Nov 03 17:34:17.718815 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.718793    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d"}
Nov 03 17:34:17.753102 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.753070    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"]
Nov 03 17:34:17.755794 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:17.755762    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a774fe_c887_4a8c_96a6_ed3b863e344c.slice/crio-33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d WatchSource:0}: Error finding container 33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d: Status 404 returned error can't find the container with id 33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d
Nov 03 17:34:17.832920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.832855    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9znfw" podStartSLOduration=2.418021061 podStartE2EDuration="3.832838503s" totalImagesPullingTime="1.321068721s" totalInitContainerRuntime="93.748721ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:17.831434547 +0000 UTC m=+66.976902141" watchObservedRunningTime="2026-11-03 17:34:17.832838503 +0000 UTC m=+66.978306092"
Nov 03 17:34:17.944002 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.942507    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5868466dd4-phngg"]
Nov 03 17:34:17.945042 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.945004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:17.955628 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:17.955576    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q" podStartSLOduration=2.262432345 podStartE2EDuration="3.955561434s" totalImagesPullingTime="1.693129089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:17.953663953 +0000 UTC m=+67.099131628" watchObservedRunningTime="2026-11-03 17:34:17.955561434 +0000 UTC m=+67.101029025"
Nov 03 17:34:18.001788 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.001695    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 17:34:18.012659 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.012585    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5868466dd4-phngg"]
Nov 03 17:34:18.044789 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.044701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.044961 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.044810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.044961 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.044850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.044961 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.044877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.044961 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.044903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.044961 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.044947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.045210 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.045014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.045210 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.045042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.146570 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.146500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.146814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.146874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.146905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.146935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.146982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.147049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.147082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.147589    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.147825    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.148350    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.148652    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.149011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.148973    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.150092 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.150072    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.160876 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.160824    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.211046 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.210975    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.259250 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.259171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:18.597372 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.597285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:18.661382 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.661353    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5868466dd4-phngg"]
Nov 03 17:34:18.666254 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:18.666209    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f705b7_0fa2_44b0_a491_479876d4282a.slice/crio-c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568 WatchSource:0}: Error finding container c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568: Status 404 returned error can't find the container with id c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568
Nov 03 17:34:18.735695 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.735583    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d"}
Nov 03 17:34:18.737884 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.737777    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5868466dd4-phngg" event={"ID":"c7f705b7-0fa2-44b0-a491-479876d4282a","Type":"ContainerStarted","Data":"c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568"}
Nov 03 17:34:18.767795 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.766434    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"]
Nov 03 17:34:18.768338 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.768270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:18.770055 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.769115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:18.828040 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.828010    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"]
Nov 03 17:34:18.828526 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.828495    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:34:18.828624 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.828584    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xlwcw\""
Nov 03 17:34:18.960953 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.960921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8vr\" (UniqueName: \"kubernetes.io/projected/d6bc2a32-53d4-4ad7-a296-74dae0153ded-kube-api-access-rv8vr\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:18.961114 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.960968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bc2a32-53d4-4ad7-a296-74dae0153ded-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:18.977506 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:18.977425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:19.062137 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.062104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bc2a32-53d4-4ad7-a296-74dae0153ded-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:19.062250 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.062228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8vr\" (UniqueName: \"kubernetes.io/projected/d6bc2a32-53d4-4ad7-a296-74dae0153ded-kube-api-access-rv8vr\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:19.066982 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.066583    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bc2a32-53d4-4ad7-a296-74dae0153ded-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:19.129196 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.129135    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8vr\" (UniqueName: \"kubernetes.io/projected/d6bc2a32-53d4-4ad7-a296-74dae0153ded-kube-api-access-rv8vr\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:19.384238 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.384151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:19.420621 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.419928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:19.591269 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.591179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:19.747937 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.746796    2579 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0" exitCode=0
Nov 03 17:34:19.747937 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.746886    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"}
Nov 03 17:34:19.751667 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.751237    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5868466dd4-phngg" event={"ID":"c7f705b7-0fa2-44b0-a491-479876d4282a","Type":"ContainerStarted","Data":"beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297"}
Nov 03 17:34:19.755336 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.755298    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vqsh" event={"ID":"634768ef-d8f7-474d-9ee4-b479a1b1f17b","Type":"ContainerStarted","Data":"50e549f125c9107f775ac524f673cb599913354636131be14f58da9b9d42e20f"}
Nov 03 17:34:19.755453 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.755359    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vqsh" event={"ID":"634768ef-d8f7-474d-9ee4-b479a1b1f17b","Type":"ContainerStarted","Data":"f92b66653a8002388d55422af6d9ab0190ae61fcedce46a49e3ba65dd1d3df75"}
Nov 03 17:34:19.778011 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:19.777985    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"]
Nov 03 17:34:19.780140 ip-10-0-139-156 kubenswrapper[2579]: W1103 17:34:19.780108    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bc2a32_53d4_4ad7_a296_74dae0153ded.slice/crio-2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5 WatchSource:0}: Error finding container 2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5: Status 404 returned error can't find the container with id 2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5
Nov 03 17:34:20.010672 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:20.010563    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8vqsh" podStartSLOduration=66.314815253 podStartE2EDuration="1m8.010542607s" totalImagesPullingTime="1.695727354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:20.008863109 +0000 UTC m=+69.154330705" watchObservedRunningTime="2026-11-03 17:34:20.010542607 +0000 UTC m=+69.156010197"
Nov 03 17:34:20.110828 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:20.110744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:20.132389 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:20.132335    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5868466dd4-phngg" podStartSLOduration=3.132317726 podStartE2EDuration="3.132317726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:20.131661303 +0000 UTC m=+69.277128896" watchObservedRunningTime="2026-11-03 17:34:20.132317726 +0000 UTC m=+69.277785357"
Nov 03 17:34:20.275055 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:20.274973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:20.763629 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:20.763591    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz" event={"ID":"d6bc2a32-53d4-4ad7-a296-74dae0153ded","Type":"ContainerStarted","Data":"2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5"}
Nov 03 17:34:21.782367 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:21.782323    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"91ba6058453fc16cdb4693a2601ea2dfe0e67819ee0f4cdfc90bea68094d83ee"}
Nov 03 17:34:21.782903 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:21.782453    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"cf7158476089c5956052a6ac919900e8ba522c3775e656d7925ac298943620c5"}
Nov 03 17:34:21.782903 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:21.782489    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"f6f9ab060c6ffcd7619b703863d1d3119b62e3d638c7c51499e84bf90229ab61"}
Nov 03 17:34:22.792747 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.792708    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"f95a7276c3b4b945dbf9eaa26a68394d6876d11a4ef762db2a1b7bbdc3ea9695"}
Nov 03 17:34:22.793161 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.792809    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"ccee6dd2dd16271e2e51e48427d1884927768b7ce589d1bc8e591b62884ee170"}
Nov 03 17:34:22.794587 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.794564    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz" event={"ID":"d6bc2a32-53d4-4ad7-a296-74dae0153ded","Type":"ContainerStarted","Data":"7e28a2e4d570e848544396e959c174622f214d47245395394cde4ac54d22e855"}
Nov 03 17:34:22.794822 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.794775    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:22.799491 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.799458    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"}
Nov 03 17:34:22.799604 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.799543    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"}
Nov 03 17:34:22.799604 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.799586    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"}
Nov 03 17:34:22.799707 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.799630    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"}
Nov 03 17:34:22.799707 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.799668    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"}
Nov 03 17:34:22.801137 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.801092    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:22.801438 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.801411    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvv9c" event={"ID":"a2260c22-12fe-411e-9a84-e8cd21d7389d","Type":"ContainerStarted","Data":"ee3a97076251880702efac48bdb7a7f378a810b96c9d43c74c6e69ffe3acedb1"}
Nov 03 17:34:22.801920 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.801902    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:22.913381 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:22.913275    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz" podStartSLOduration=2.553722563 podStartE2EDuration="4.913258657s" totalImagesPullingTime="2.359536094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:22.912875524 +0000 UTC m=+72.058343135" watchObservedRunningTime="2026-11-03 17:34:22.913258657 +0000 UTC m=+72.058726252"
Nov 03 17:34:23.031000 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.030949    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tvv9c" podStartSLOduration=67.17706395 podStartE2EDuration="1m12.030930928s" totalImagesPullingTime="4.853866978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:23.030278243 +0000 UTC m=+72.175745836" watchObservedRunningTime="2026-11-03 17:34:23.030930928 +0000 UTC m=+72.176398521"
Nov 03 17:34:23.646761 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.646699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:23.809902 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.809856    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cz52j" event={"ID":"46723e8e-0ca8-4209-9904-b35885a2f90f","Type":"ContainerStarted","Data":"bbd698f026385a6b705668548ae152db4c3f892de45978d08735898e9a0ea683"}
Nov 03 17:34:23.813435 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.813405    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd" event={"ID":"25a774fe-c887-4a8c-96a6-ed3b863e344c","Type":"ContainerStarted","Data":"563dd6c63417b81fff5b4e5f11bb101570143f544cf3421bff2b6e4846c58fb4"}
Nov 03 17:34:23.813591 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.813549    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:23.817013 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.816984    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerStarted","Data":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"}
Nov 03 17:34:23.927144 ip-10-0-139-156 kubenswrapper[2579]: I1103 17:34:23.927089    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cz52j" podStartSLOduration=65.271531868 podStartE2EDuration="1m11.927074325s" totalImagesPullingTime="6.655542457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:23.926642078 +0000 UTC m=+73.072109670" watchObservedRunningTime="2026-11-03 17:34:23.927074325 +0000 UTC m=+73.072541917"
Nov 03 17:34:23.931652 ip-10-0-139-156 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 17:34:23.943125 ip-10-0-139-156 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 17:34:23.943297 ip-10-0-139-156 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 17:34:23.943351 ip-10-0-139-156 systemd[1]: kubelet.service: Consumed 4.742s CPU time.
Nov 03 17:34:23.951520 ip-10-0-139-156 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:34:24.020839 ip-10-0-139-156 kubenswrapper[5838]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:34:24.020839 ip-10-0-139-156 kubenswrapper[5838]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:34:24.020839 ip-10-0-139-156 kubenswrapper[5838]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:34:24.020839 ip-10-0-139-156 kubenswrapper[5838]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:34:24.022928 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022842    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.022928 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022928    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.022928 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022932    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022936    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022940    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022943    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022946    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022948    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022951    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022954    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022957    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022959    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022962    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022964    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022967    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022969    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022972    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022974    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022977    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022980    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022982    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022985    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.023027 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022995    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.022998    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023000    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023002    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023005    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023007    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023010    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023012    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023015    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023017    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023020    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023022    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023025    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023028    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023031    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023033    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023036    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023039    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023042    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023045    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.023509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023048    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023051    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023053    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023056    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023058    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023061    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023064    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023066    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023069    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023072    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023075    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023077    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023080    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023082    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023086    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023090    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023093    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023096    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023099    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.024008 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023101    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023104    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023106    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023109    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023112    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023114    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023117    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023119    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023122    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023125    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023128    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023132    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023134    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023137    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023139    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023142    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023144    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023147    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023153    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.024470 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023156    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023158    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023161    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023164    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023166    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023168    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023171    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023174    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023177    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023180    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023183    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023185    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023188    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023191    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023193    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023196    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023198    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023201    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023203    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.024931 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023206    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023210    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023214    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023218    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023222    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023225    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.023229    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024043    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024050    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024054    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024057    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024059    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024064    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024067    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024070    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024072    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024075    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024078    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024080    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.025374 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024083    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024086    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024089    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024092    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024094    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024097    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024101    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024103    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024106    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024109    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024111    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024114    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024116    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024119    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024121    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024124    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024126    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024129    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024132    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024135    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.025833 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024138    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024141    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024143    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024146    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024148    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024152    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024155    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024157    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024161    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024165    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024167    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024170    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024172    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024175    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024178    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024181    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024185    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024188    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024191    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024194    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.026305 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024197    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024199    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024202    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024204    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024207    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024209    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024212    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024214    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024217    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024219    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024222    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024224    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024227    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024230    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024232    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024235    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024237    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024242    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024244    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024247    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.026798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024250    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024252    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024254    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024257    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024260    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024262    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024265    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024267    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024270    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024273    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024275    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024278    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024280    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024283    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024285    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024288    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024290    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024293    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024295    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.027270 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024298    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024301    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024303    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024306    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024308    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024310    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024313    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024324    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024327    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024330    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024334    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024336    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024339    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024344    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024348    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024951    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024960    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024964    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024967    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.027769 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024971    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024974    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024976    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024979    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024981    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024984    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024987    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024990    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024992    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024995    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.024997    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025000    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025003    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025005    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025008    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025010    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025013    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025015    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025018    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025020    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.028246 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025023    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025026    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025029    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025033    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025036    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025038    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025041    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025043    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025046    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025049    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025051    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025054    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025075    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025080    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025084    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025086    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025089    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025092    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025095    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.028777 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025098    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025100    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025102    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025105    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025113    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025116    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025119    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025121    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025124    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025126    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025129    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025132    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025134    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025137    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025145    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025148    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025153    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025156    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025159    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.029249 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025162    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025165    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025167    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025171    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025174    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025176    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025179    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025181    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025184    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025186    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025189    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025191    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025194    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025196    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025199    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025201    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025204    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025206    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025209    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.029708 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025211    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025214    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025216    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025219    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025221    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025224    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025226    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025228    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025231    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025241    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025245    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025248    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025250    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025252    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025256    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025259    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025264    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025266    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025269    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.030184 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025271    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025274    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025277    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025279    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025282    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.025284    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025378    5838 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025386    5838 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025392    5838 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025397    5838 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025402    5838 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025405    5838 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025410    5838 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025414    5838 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025417    5838 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025420    5838 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025423    5838 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025427    5838 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025430    5838 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025433    5838 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025436    5838 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025439    5838 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025442    5838 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:34:24.030707 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025447    5838 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025452    5838 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025455    5838 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025458    5838 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025461    5838 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025464    5838 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025470    5838 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025473    5838 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025476    5838 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025480    5838 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025483    5838 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025486    5838 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025489    5838 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025492    5838 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025495    5838 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025499    5838 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025502    5838 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025504    5838 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025507    5838 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025510    5838 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025513    5838 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025539    5838 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025542    5838 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025545    5838 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:34:24.031275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025549    5838 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025552    5838 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025556    5838 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025559    5838 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025562    5838 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025565    5838 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025568    5838 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025571    5838 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025574    5838 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025578    5838 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025581    5838 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025584    5838 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025587    5838 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025591    5838 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025595    5838 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025599    5838 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025602    5838 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025605    5838 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025608    5838 flags.go:64] FLAG: --help="false"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025611    5838 flags.go:64] FLAG: --hostname-override="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025614    5838 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025617    5838 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025621    5838 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:34:24.031852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025624    5838 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025627    5838 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025630    5838 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025633    5838 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025636    5838 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025639    5838 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025641    5838 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025644    5838 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025647    5838 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025650    5838 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025653    5838 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025656    5838 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025659    5838 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025662    5838 flags.go:64] FLAG: --lock-file=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025665    5838 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025668    5838 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025671    5838 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025676    5838 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025679    5838 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025685    5838 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025688    5838 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025690    5838 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025694    5838 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025698    5838 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025701    5838 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:34:24.032468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025706    5838 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025709    5838 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025712    5838 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025715    5838 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025718    5838 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025721    5838 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025724    5838 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025748    5838 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025751    5838 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025754    5838 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025766    5838 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025769    5838 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025772    5838 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025775    5838 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025778    5838 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025781    5838 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025785    5838 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025787    5838 flags.go:64] FLAG: --port="10250"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025790    5838 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025793    5838 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0566d144036bbe524"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025797    5838 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025800    5838 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025803    5838 flags.go:64] FLAG: --register-node="true"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025805    5838 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025809    5838 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025812    5838 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:34:24.033430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025815    5838 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025819    5838 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025823    5838 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025827    5838 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025832    5838 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025835    5838 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025838    5838 flags.go:64] FLAG: --runonce="false"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025841    5838 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025845    5838 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025848    5838 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025851    5838 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025855    5838 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025857    5838 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025861    5838 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025864    5838 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025866    5838 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025869    5838 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025872    5838 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025875    5838 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025878    5838 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025881    5838 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025884    5838 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025889    5838 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025892    5838 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025895    5838 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:34:24.034385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025899    5838 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025902    5838 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025905    5838 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025908    5838 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025911    5838 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025914    5838 flags.go:64] FLAG: --v="2"
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025918    5838 flags.go:64] FLAG: --version="false"
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025922    5838 flags.go:64] FLAG: --vmodule=""
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025927    5838 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.025931    5838 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026101    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026107    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026110    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026113    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026115    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026118    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026120    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026123    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026126    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026128    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026131    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.035090 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026133    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026136    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026139    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026141    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026144    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026147    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026149    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026152    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026155    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026157    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026160    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026162    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026165    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026167    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026170    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026172    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026175    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026178    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026180    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026183    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.035587 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026185    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026189    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026192    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026195    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026197    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026200    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026202    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026205    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026207    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026210    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026212    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026215    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026217    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026220    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026222    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026225    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026227    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026230    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026233    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026235    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.036067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026238    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026240    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026242    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026246    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026250    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026253    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026257    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026260    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026263    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026266    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026268    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026271    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026273    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026278    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026281    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026283    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026286    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026288    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026291    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.036553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026293    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026296    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026298    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026301    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026303    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026306    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026309    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026312    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026314    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026317    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026319    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026322    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026324    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026327    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026330    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026332    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026334    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026337    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026339    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.037009 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026342    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026344    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026347    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026349    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026352    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026354    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026356    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026361    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026364    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026372    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026375    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026378    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026380    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026383    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026385    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026388    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.037453 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026391    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.026417    5838 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026460    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026464    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026467    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026470    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026472    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026475    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026477    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026480    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026482    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026485    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026487    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026490    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026492    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026495    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.037851 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026497    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026501    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026504    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026506    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026509    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026511    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026514    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026516    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026520    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026522    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026533    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026536    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026538    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026541    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026543    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026546    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026548    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026551    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026553    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026556    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.038229 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026558    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026562    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026566    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026569    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026572    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026575    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026578    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026582    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026584    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026586    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026589    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026592    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026594    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026598    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026600    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026603    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026605    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026608    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.038881 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026610    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026613    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026616    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026619    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026622    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026630    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026632    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026635    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026637    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026640    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026642    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026645    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026647    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026650    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026653    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026655    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026657    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026660    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026662    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.039410 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026665    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026667    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026669    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026672    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026674    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026677    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026679    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026682    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026686    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026688    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026699    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026702    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026704    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026707    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026709    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026713    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026716    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026718    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026721    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026741    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.039968 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026744    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026747    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026749    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026752    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026755    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026757    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026760    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026762    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026764    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026767    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026770    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026772    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026776    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026778    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.026781    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.031891    5838 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.031911    5838 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.031929    5838 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.031934    5838 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:34:24.040634 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032012    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032018    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032022    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032025    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032028    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032031    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032034    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032036    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032039    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032042    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032044    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032047    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032049    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032052    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032055    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032057    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032060    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032062    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032064    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.041487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032067    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032070    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032072    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032075    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032077    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032080    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032083    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032086    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032088    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032091    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032100    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032103    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032106    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032109    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032111    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032114    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032116    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032118    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032121    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032123    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.042203 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032126    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032134    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032137    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032140    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032143    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032146    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032150    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032154    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032157    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032160    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032163    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032166    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032169    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032172    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032174    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032177    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032179    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032182    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032184    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.042907 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032187    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032189    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032192    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032194    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032202    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032205    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032209    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032214    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032218    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032222    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032226    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032230    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032235    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032239    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032243    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032246    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032249    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032251    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032254    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032257    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.043575 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032259    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032262    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032265    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032267    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032270    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032272    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032275    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032277    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032282    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032285    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032288    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032290    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032293    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032295    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032298    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032300    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032303    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032313    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032326    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.044213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032329    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032332    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032334    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032337    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032339    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032343    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032346    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032349    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.032351    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.032383    5838 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.033371    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.033389    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034254    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034379    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034385    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034389    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.044677 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034392    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034396    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034402    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034407    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034412    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034416    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034419    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034421    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034424    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034427    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034430    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034433    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034436    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034439    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034441    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034444    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034447    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034449    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034452    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034454    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.045163 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034457    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034459    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034462    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034464    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034467    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034469    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034473    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034478    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034488    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034491    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034494    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034496    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034499    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034504    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034510    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034513    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034516    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034519    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034523    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.045656 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034527    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034530    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034533    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034536    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034538    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034541    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034544    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034546    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034548    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034551    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034554    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034556    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034559    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034561    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034564    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034566    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034569    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034572    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034575    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034577    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.046250 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034580    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034583    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034586    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034588    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034593    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034597    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034600    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034602    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034605    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034608    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034610    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034613    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034616    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034618    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034621    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034624    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034626    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034629    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034632    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.046876 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034634    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034637    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034640    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034642    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034645    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034647    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034649    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034652    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034654    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034657    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034659    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034662    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034664    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034667    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034669    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034671    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034674    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034676    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034679    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034681    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.047514 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034684    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034687    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.034723    5838 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034865    5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034872    5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034875    5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034878    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034880    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034883    5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034885    5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034888    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034891    5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034899    5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034902    5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034905    5838 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:34:24.048192 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034907    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034912    5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034916    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034920    5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034922    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034925    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034928    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034930    5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034933    5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034936    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034938    5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034941    5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034944    5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034946    5838 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034949    5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034951    5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034954    5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034956    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034959    5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034961    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:34:24.048694 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034964    5838 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034966    5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034969    5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034972    5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034975    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034977    5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034980    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034982    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034985    5838 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034987    5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.034995    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035000    5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035003    5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035006    5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035009    5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035013    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035015    5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035018    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035021    5838 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:34:24.049422 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035023    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035026    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035028    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035030    5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035033    5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035035    5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035038    5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035040    5838 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035043    5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035045    5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035047    5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035050    5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035052    5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035055    5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035057    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035060    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035062    5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035065    5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035068    5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035070    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:34:24.050020 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035073    5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035075    5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035077    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035080    5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035089    5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035092    5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035095    5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035097    5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035104    5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035107    5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035110    5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035112    5838 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035115    5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035117    5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035120    5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035123    5838 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035125    5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035128    5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035130    5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035133    5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:34:24.050559 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035136    5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035138    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035140    5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035143    5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035147    5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035150    5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035152    5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035155    5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035158    5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035160    5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035163    5838 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035165    5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035168    5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035170    5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:34:24.035173    5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.035238    5838 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.036337    5838 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.036398    5838 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.036594    5838 server.go:1014] "Starting client certificate rotation"
Nov 03 17:34:24.051064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.036741    5838 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.036796    5838 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:28:11 +0000 UTC" deadline="2028-07-16 18:22:16.409929588 +0000 UTC"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.036824    5838 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14904h47m52.373107834s"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.037082    5838 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.038357    5838 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.039624    5838 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.042215    5838 log.go:25] "Validated CRI v1 image API"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.043253    5838 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:34:24.051521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.045794    5838 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9af7367f-55c7-4331-ab17-3d39fc8e8584:/dev/nvme0n1p3 9fbbc0f0-f8a8-418c-9998-5caa8607dc0c:/dev/nvme0n1p4]
Nov 03 17:34:24.051813 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.045817    5838 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5/userdata/shm major:0 minor:420 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d/userdata/shm major:0 minor:391 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07057c45-a937-446c-96ea-c3e74244776b/volumes/kubernetes.io~projected/kube-api-access-62slh:{mountpoint:/var/lib/kubelet/pods/07057c45-a937-446c-96ea-c3e74244776b/volumes/kubernetes.io~projected/kube-api-access-62slh major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07057c45-a937-446c-96ea-c3e74244776b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/07057c45-a937-446c-96ea-c3e74244776b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~projected/kube-api-access-mgdz2:{mountpoint:/var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~projected/kube-api-access-mgdz2 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~projected/kube-api-access-ddhm9:{mountpoint:/var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~projected/kube-api-access-ddhm9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d/volumes/kubernetes.io~projected/kube-api-access-xq5j5:{mountpoint:/var/lib/kubelet/pods/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d/volumes/kubernetes.io~projected/kube-api-access-xq5j5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~projected/kube-api-access-kqzwx:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~projected/kube-api-access-kqzwx major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~projected/kube-api-access-dcdbn:{mountpoint:/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~projected/kube-api-access-dcdbn major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/alertmanager/9 major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/config-reloader/6 major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~empty-dir/config-out major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~projected/kube-api-access-s4h5z:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~projected/kube-api-access-s4h5z major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~projected/tls-assets major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/config-volume major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/web-config major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46723e8e-0ca8-4209-9904-b35885a2f90f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/46723e8e-0ca8-4209-9904-b35885a2f90f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b47b5da-225b-4e79-aab7-f81b12aa6191/volumes/kubernetes.io~projected/kube-api-access-r9bqd:{mountpoint:/var/lib/kubelet/pods/4b47b5da-225b-4e79-aab7-f81b12aa6191/volumes/kubernetes.io~projected/kube-api-access-r9bqd major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volumes/kubernetes.io~projected/kube-api-access-jc4rf:{mountpoint:/var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volumes/kubernetes.io~projected/kube-api-access-jc4rf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/634768ef-d8f7-474d-9ee4-b479a1b1f17b/volumes/kubernetes.io~projected/kube-api-access-btmj5:{mountpoint:/var/lib/kubelet/pods/634768ef-d8f7-474d-9ee4-b479a1b1f17b/volumes/kubernetes.io~projected/kube-api-access-btmj5 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/634768ef-d8f7-474d-9ee4-b479a1b1f17b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/634768ef-d8f7-474d-9ee4-b479a1b1f17b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6544fcbf-d2a3-48fc-8f79-71cc649dd82b/volumes/kubernetes.io~projected/kube-api-access-jlwgl:{mountpoint:/var/lib/kubelet/pods/6544fcbf-d2a3-48fc-8f79-71cc649dd82b/volumes/kubernetes.io~projected/kube-api-access-jlwgl major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d4951c9-ab34-4727-9272-fc331a7fae42/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7d4951c9-ab34-4727-9272-fc331a7fae42/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f48d878-d642-49c6-a4db-6e74ea2542fc/volumes/kubernetes.io~projected/kube-api-access-8d4p6:{mountpoint:/var/lib/kubelet/pods/7f48d878-d642-49c6-a4db-6e74ea2542fc/volumes/kubernetes.io~projected/kube-api-access-8d4p6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e659c14-b9ea-4485-a903-eb9ed8e42272/volumes/kubernetes.io~projected/kube-api-access-6nv6s:{mountpoint:/var/lib/kubelet/pods/9e659c14-b9ea-4485-a903-eb9ed8e42272/volumes/kubernetes.io~projected/kube-api-access-6nv6s major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2260c22-12fe-411e-9a84-e8cd21d7389d/volumes/kubernetes.io~projected/kube-api-access-zkzb4:{mountpoint:/var/lib/kubelet/pods/a2260c22-12fe-411e-9a84-e8cd21d7389d/volumes/kubernetes.io~projected/kube-api-access-zkzb4 major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~projected/kube-api-access-jwfjm:{mountpoint:/var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~projected/kube-api-access-jwfjm major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5739357-c9b8-445d-9b36-137395fd0857/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/c5739357-c9b8-445d-9b36-137395fd0857/volumes/kubernetes.io~secret/tls-certificates major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~projected/kube-api-access-vq9dm:{mountpoint:/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~projected/kube-api-access-vq9dm major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~projected/kube-api-access-snwrp:{mountpoint:/var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~projected/kube-api-access-snwrp major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6bc2a32-53d4-4ad7-a296-74dae0153ded/volumes/kubernetes.io~projected/kube-api-access-rv8vr:{mountpoint:/var/lib/kubelet/pods/d6bc2a32-53d4-4ad7-a296-74dae0153ded/volumes/kubernetes.io~projected/kube-api-access-rv8vr major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6bc2a32-53d4-4ad7-a296-74dae0153ded/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d6bc2a32-53d4-4ad7-a296-74dae0153ded/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb54bc0-f60d-4745-bcfe-5978119c0bde/volumes/kubernetes.io~projected/kube-api-access-2dgmd:{mountpoint:/var/lib/kubelet/pods/dbb54bc0-f60d-4745-bcfe-5978119c0bde/volumes/kubernetes.io~projected/kube-api-access-2dgmd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d/volumes/kubernetes.io~projected/kube-api-access-59tph:{mountpoint:/var/lib/kubelet/pods/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d/volumes/kubernetes.io~projected/kube-api-access-59tph major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~projected/kube-api-access-xv6tv:{mountpoint:/var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~projected/kube-api-access-xv6tv major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0481c4-d60c-47f9-9fc5-901a405928bc/volumes/kubernetes.io~projected/kube-api-access-wprfg:{mountpoint:/var/lib/kubelet/pods/ea0481c4-d60c-47f9-9fc5-901a405928bc/volumes/kubernetes.io~projected/kube-api-access-wprfg major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0481c4-d60c-47f9-9fc5-901a405928bc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ea0481c4-d60c-47f9-9fc5-901a405928bc/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9d1d22ff1436292f229131a4a9c167cd6c9ad3c67b3ffc130dbb0fbc2cc3361f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1f3779606431bb0efb2a7eaa9053fba077c24f40312dcf48323f4b3667d7008f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7f396a40e343befb621b036790682d6074bc37a61b17f837912ac7aeb97e4723/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d3b820fc9270ee9031fbcc7c0cb8a60086cee17a1591df08eefbb1d31a72510a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d7b53b3c0a96c3ee5442bd7c8ec0223409f9c29a6240175cd186ae5e835374b3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/efefa80f4e4eac02a54051cfc8a530dee5ab32727bc83886ce2d6218591f9253/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/df94b2e2f71a87be035cb128770b9d5474f00a2c8f2cf3ecf77b67bfa3a05441/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/72ab669093140ca64b0a39b205b4907670c1f9f2cd7575587b07a7cedc4dbb3e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d244530d4f59fce28ea67f90b177672a9dee8f586c5eef8fa4322e63144e48f8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ef9041075486c5e65ff353ab22e6f532c62a6c0f4e691aa46e100f003181fe09/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c623eb918f51ba13e52c4557fe282a07d4f6027c71132df4dd23cc058e4f5dea/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/51984e83385d3f7e5091aa8d8995d4274a5714936a39f5c57bd62463299c90d0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b35d6d57a111c1246ea13143d3c1c3215991bc2acb32b43366277f484661a180/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a9d24684d466fd71a576579a53e18e9e07216bc12f3d232b32191897bd2d8bcc/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/efa3a1809979661cfa1d3365c7f2679734f8bb42098b8dbbfab48599e786065e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/949ee81549e03381c798951551200900808669035c1875113a62567b56e8e124/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2f3c4276bfa12dd6f46c479e137b42da3bca899995c9f195dff44914fea04e10/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6510a676a32b9e335350c46a2272eb77da9efcce75e886276edeb9cae731d357/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c7049ab3f38e987c635274dfff7699b7f79874b5f30d917331857fde1360ac40/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/af3eff03d8cebead0c7fd467bc98fa469de66c68d6bb62ee197cab5e30aecd01/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ca28413e33dae5aad5d91695ac92decc99d824c399c13904df446025a9c84e8c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/a14e860cd62ee783ff56f7e0e54d9838ac1e3f6d714b070d0aaa3784c40c3a77/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/f4444dc40592fe1c3a5b43ac7d49cc27ca27e191b2bac81b6ee929d4fa0ef426/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/a769d5a564c24e8a4c3203e5b75de59f6b0a843851d3e0f0da671c2c4830d7b7/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/b9a3c60eb262ba7414431fb2692a17c2fa514a7cc5910f88f29303c14286e723/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/59c320ac55b388d3bd2f124d93418b7426003cb689f9a6b8585e452ca6007112/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/ec84fbd55aa7d9543cf3d5fb807d811e7e1977e96cc77974c0392039f002f351/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4220d25450e9f90ca38047bae50ad4ded53ce84c4483940d420dfcf9069eb9e9/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/127285471e2e131497b6da77fb089ece169554ccc4c350a0d1de4dc58e03e773/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/f55f77bd7bf40f7b8407f5735aa1e12824db53e50b4909a172c043e95a7d4002/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/cb458da28c5fa98c4441b9bd7f2f4bedd502a560400989f7f43da1a28c1d6155/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/7d04f993140e594f6f8c486994168dbe9932d5e956bd5494781730c0d31ce13c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/613c04623fdc9778899e947f63396235951d41d8fc2ce4fe3d5c16ad533145ed/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/ad9b0248a2fd44aacdd7540429f175619a2d1b285a69bc5fa1caf1b62a09f9f5/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/1daf2e201c703ea930cb848c985e8692d8b1252b2f840d10b544c96d9971fe39/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/434f85cd0b1ff723458b0d55f702ec7df8dbd05913de6dbafa6a1a95ee3ae66b/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/aeca85b00d223bf5d234264d3665489b486eb1c415e0b8533c6e42747a867e19/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/0e4125046e6362da42c0eda2be554dc787a080a007e434aedea3dfdd1073917d/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/3fafb87d60fb25bdbe2dc0320cc243e4bfd4afceb467162757bcc1b0cec60cc3/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/f600bce80644075118721446bd59f53f4bed055572cf04b90c7cee0bbdf05a15/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/95e039ea20c684c3feddd90ad2bea3949b7ef65e76be5db6c55711d4761a2958/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/d6e37af1b32ad0087af3acf62a32455b75732f1a149584b0614704eba67119cc/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/14caf49c4775908e71f8d9a1b252fbcc8928b943bf125e8369d579e5a8551e3f/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/939c0d7ce321e9388fec5cd7e74b0804dac2e2564602a0f436d2077820ef71ec/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/78fd21fc22933bb9569b68da58f495d5c509588644586f93c130165ff0994462/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/b476979379b130ca71f8b75aacf62dbc3ad97c37ad19710e5e798115a6b735f5/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/17570e7a892f5917eb7a17598fffbdfc4b4a44be35a36f28feb84dac176b8b2e/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/6edc3388f55a83703c3b5a7da32b2ea189d2b9e42cbe5437a80f215c6122db89/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/80eb4772c82dffba44fa8f0c64613d8137b8e4a39fbd87156f9b888a88d47aba/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/325494296ee75d20509ca560aea42408da61688489756cf88f4be22561cbe442/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/dc88dabeda420e1756a792fca59d73a47210803a9496ed9b3b2e1de3591b4717/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/1253691c1bd4b60695b1b0decc79f271890202ac8df2f2985ff4c1f05751e1e9/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/1aa8dc9f5a290949afa1500552f4a2bccf10d3145a9ce62e394f3fdc729753f7/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/baa295989a7673fa1ae3a4cb23d429c2083429232cace82332945e60a98324d3/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/a657867d182aad7c77adee3e68cbe15742f99c97c5f820160a06c3a6f8c0132e/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/0554906f73260def162ad45cc15797e74d80be2fc018baca573c2f20990a5ceb/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/ba77a703c4cb752a6d8e40a1ee3bea7eaa741ecfd154e10ae1d661cd49e03811/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/c11386475508b557a6e2900cc836f35b0d09b1602b5c40e57509018fa01c82d4/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/93f47e5b6e904bda7f47291d8bac18a970f0845d2d878ac8766c6de822911ede/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/a54b666661d00ed34f0abe08595d6f1eef72e47b00ed1b5974e30af38b2399d8/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/260fac26c244fe79634a4262a3deb357a770317bc9f7d791f35418066476021d/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/166cb24cd13622bcb49454ad4f9b660fea869c1078cd0b8c7dfad33390162930/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/0f9210011d67d0419e4957b35064267139de1c2bbb7058597a41060b01ea8243/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/6843d7471059eaa9d839417c1ccc7141fbdd62ff6636beda01b76b0a73f278cd/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/c32cb64303e176077967d0cd426e40235f7bb3aa26febe881fbace1a0b3036dd/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/4e6a05dfae20eae5336b1f8d6fbf22a6c4a2ae9720669d60474b87b2000e0d6e/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/efa4cacb9ac4c30b64ac591359fc3bc11100087515bd0b3f1e103f26e97699ef/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/e4182f7696cd93a1da17f70746c71f9ce0f959c3424cceef6f8b1303e2254d5d/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/6285dbd79927a629559ae2a68b49b7ec104507f0e750d33e2e70a0cd1d5aeb90/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/201124b6b9e91f87a2b90674ae3aaab2bd34d202f0facf78bd563e4eb917e632/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/3c218de77cd8482a2e52c3e1e1c1f9cf376aed548ff5afb6042a7d4c06e6f95b/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/61a6e922bf651761da7b92d9caecd0007f0f5e204faaec630da46924c8ef45b3/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/bd3d9411dc3979d155920541a2247b21e064d6cff7e51ce4b1e05ddf47f8106c/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-65:{mountpoint:/var/lib/containers/storage/overlay/0a6a711679403223485b70009cc548b2248b4ee9d34837443c9383dd6fdc261a/merged major:0 minor:65 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/177c6042c6183a15e5d59de3ed29912fbb06ce3120ded72f7824729f8a775c70/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/67a2b1da8a17d2caec41f90be568d67c02cd24355158976dd3681c8e00258714/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/2f408a3b1d68f8e6c5f86c6cdb1fcd06bb301259c0222ab03010141c7b90783c/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b14aeb53c481a0a5c793901c9013f0e8abf328cdd41109a6a326fbeffa39804e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/06db4b639fa36d5e8983ac8ff3cdb382a71f4c5b7e9dcbe222233ad69af0d3c3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e31ace11cc433c8a5700abbdbafdb25b76f80fe71d049abcd9dd7e3ec7f8fcce/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/5f4a7ff6443539d4866eb64d57f36331e1fb9c40a67201ef9e3665f93ec514e0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/03a2eb236fca8e90bb4b7766ea39dffc49a650801e55c6caa93f796f5c843af7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/33cc5ec4888fdf3e3b7c10c466fa5ff63734dff8fb0b8b4e19592e69d56483bf/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 03 17:34:24.056634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.056164    5838 manager.go:217] Machine: {Timestamp:2026-11-03 17:34:24.055640594 +0000 UTC m=+0.066378657 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099494 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279ce56d4e03e21aa1eeb43fe4790a SystemUUID:ec279ce5-6d4e-03e2-1aa1-eeb43fe4790a BootID:c1774ac9-5844-480f-90e2-4ded4c89403f Filesystems:[{Device:/var/lib/kubelet/pods/07057c45-a937-446c-96ea-c3e74244776b/volumes/kubernetes.io~projected/kube-api-access-62slh DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~projected/kube-api-access-xv6tv DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5a05fb58525b7eb543a5b35cb466c0e4c684cda38bae5d52032158f0c993711/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4e29354fe169fed75656848164000f0bc6d2041e17e13c7983ead08da2710d8/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-65 DeviceMajor:0 DeviceMinor:65 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~projected/kube-api-access-s4h5z DeviceMajor:0 DeviceMinor:362 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6bc2a32-53d4-4ad7-a296-74dae0153ded/volumes/kubernetes.io~projected/kube-api-access-rv8vr DeviceMajor:0 DeviceMinor:427 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f48d878-d642-49c6-a4db-6e74ea2542fc/volumes/kubernetes.io~projected/kube-api-access-8d4p6 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c5739357-c9b8-445d-9b36-137395fd0857/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:350 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/749d47e8e90f014be8438ff5e2f296c2c8d73f2873e75898b40f9657d486002e/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7a7c16ef9d6d4abf38a3ec6cdb062a209e28877837d33e956963b0d3b040360/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:380 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0481c4-d60c-47f9-9fc5-901a405928bc/volumes/kubernetes.io~projected/kube-api-access-wprfg DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volumes/kubernetes.io~projected/kube-api-access-jc4rf DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d843061be3bf61ad86bdcc6129c2f0cb5e6ec760806fa1f8205a5494b4d6855b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76bcd32827db6947b1de1237376f979634fc1bf1d81b661856707edbc925ad40/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07057c45-a937-446c-96ea-c3e74244776b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/203de703ee98f6593a466daafa588c6171ca561044ccb91dd74f6e218335f83a/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb54bc0-f60d-4745-bcfe-5978119c0bde/volumes/kubernetes.io~projected/kube-api-access-2dgmd DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d36e6b92b8f4f5b511c0b9930e823ed9414d7b8c830a8622480b1788a79596b/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~projected/kube-api-access-mgdz2 DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5908cbf5-2e94-4a1c-9c81-f7adfd472da0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~projected/kube-api-access-snwrp DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33663e9e19c45130f42db253c7d814b8f283b9ad7c5480a599aa10f29365df1d/userdata/shm DeviceMajor:0 DeviceMinor:391 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:396 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e659c14-b9ea-4485-a903-eb9ed8e42272/volumes/kubernetes.io~projected/kube-api-access-6nv6s DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/336e60fbe30c8b7d378337094a171636bfc5ba52c5c2b73c0f54d6c153d96d2f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d/volumes/kubernetes.io~projected/kube-api-access-59tph DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/634768ef-d8f7-474d-9ee4-b479a1b1f17b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:382 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/751aa46aa154ec7a14f3ba204dc7e43b99dd184f50bcdea2d9b4aa938a75ff76/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4b47b5da-225b-4e79-aab7-f81b12aa6191/volumes/kubernetes.io~projected/kube-api-access-r9bqd DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d86e54121b0db7e07bb745b032ce4bbc30ccf709c8afd2faa0d69d12f99a37c8/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/335d5be6d801164e93756603bbabef44bed68f1b48bd41b544f6592ddef12aeb/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0481c4-d60c-47f9-9fc5-901a405928bc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10fe28f8dbbacd696d51060247f5edbe517cc55f8da468a8b473c3229ca46f60/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:395 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c15aff4e33f5b2a718107473b1ed84b6348751f194117601aa4186dbb5b8d127/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd2c77f6b5a8ed7c2462f50fe5d1639818c99f40dbc6efc649b3a7ff4d4c6342/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6bc2a32-53d4-4ad7-a296-74dae0153ded/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:416 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d/volumes/kubernetes.io~projected/kube-api-access-xq5j5 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7d4951c9-ab34-4727-9272-fc331a7fae42/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb2852e1-7207-4d27-a78e-ab6a06a425c3/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df9489d-1992-48e2-9fca-b6a83d7a0cff/volumes/kubernetes.io~projected/kube-api-access-ddhm9 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~projected/kube-api-access-jwfjm DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c628272-7a16-4897-af12-59bcfedd74cb/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0ec8484fbff7afe04a1ae892df2523d5d818d4d4904735375861bba5e505fc1/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:381 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~projected/kube-api-access-kqzwx DeviceMajor:0 DeviceMinor:386 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes/kubernetes.io~projected/kube-api-access-vq9dm DeviceMajor:0 DeviceMinor:397 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b48e299b7df62c61efa865f0d1ab3b70bef41b06acde9b85720b5b8902dc3f5/userdata/shm DeviceMajor:0 DeviceMinor:420 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6544fcbf-d2a3-48fc-8f79-71cc649dd82b/volumes/kubernetes.io~projected/kube-api-access-jlwgl DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/634768ef-d8f7-474d-9ee4-b479a1b1f17b/volumes/kubernetes.io~projected/kube-api-access-btmj5 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a27e616e603d951cdcf9b48faed78d0fc467d8836778702bca377f41b57352e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c4d0a499-8879-4147-b92b-b81d727ddff3/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46723e8e-0ca8-4209-9904-b35885a2f90f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ca80f574b16dc805727a17841373148e7245d49f4cfb7f8344c898e040e50d0/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49dfd3b4ce53a3f6facb305f868cae511513428368b4032e19d1d165f5d93ef2/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a774fe-c887-4a8c-96a6-ed3b863e344c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:385 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:256 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes/kubernetes.io~projected/kube-api-access-dcdbn DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a2260c22-12fe-411e-9a84-e8cd21d7389d/volumes/kubernetes.io~projected/kube-api-access-zkzb4 DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:10fe28f8dbbacd6 MacAddress:12:47:6b:a0:4b:82 Speed:10000 Mtu:8901} {Name:203de703ee98f65 MacAddress:26:64:c9:24:cd:b8 Speed:10000 Mtu:8901} {Name:2b48e299b7df62c MacAddress:46:09:71:13:3d:5e Speed:10000 Mtu:8901} {Name:335d5be6d801164 MacAddress:aa:74:b2:66:fd:3b Speed:10000 Mtu:8901} {Name:33663e9e19c4513 MacAddress:66:55:9f:2d:77:57 Speed:10000 Mtu:8901} {Name:336e60fbe30c8b7 MacAddress:e2:1b:b8:6a:a5:d0 Speed:10000 Mtu:8901} {Name:49dfd3b4ce53a3f MacAddress:92:0c:be:32:06:97 Speed:10000 Mtu:8901} {Name:4bf1739aa502e67 MacAddress:82:18:19:9b:85:76 Speed:10000 Mtu:8901} {Name:749d47e8e90f014 MacAddress:36:e8:1f:78:6e:4b Speed:10000 Mtu:8901} {Name:751aa46aa154ec7 MacAddress:ee:95:18:1c:0c:0d Speed:10000 Mtu:8901} {Name:76bcd32827db694 MacAddress:4e:12:bf:5b:c7:13 Speed:10000 Mtu:8901} {Name:9b322cb8d668b17 MacAddress:1e:6b:95:31:d0:83 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0f:c6:3b:56:e5 Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:84:22:a2:c0:45 Speed:0 Mtu:8901} {Name:c519d15b3ce0fb1 MacAddress:76:e5:8d:e5:d4:52 Speed:10000 Mtu:8901} {Name:c7a7c16ef9d6d4a MacAddress:b2:1c:64:ed:c7:a5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:0f:c6:3b:56:e5 Speed:0 Mtu:9001} {Name:f0ec8484fbff7af MacAddress:fa:77:1c:e7:36:57 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:86:26:55:b2:71:48 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:7a:c8:9c:96:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:34:24.056634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.056637    5838 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:34:24.056976 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.056689    5838 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:34:24.057008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.056963    5838 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:34:24.057123 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.056989    5838 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-156.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:34:24.057164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057125    5838 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:34:24.057164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057133    5838 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:34:24.057164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057146    5838 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:34:24.057241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057168    5838 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:34:24.057284 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057276    5838 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:34:24.057450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057440    5838 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:34:24.057565 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057556    5838 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:34:24.057601 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057593    5838 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:34:24.057669 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057605    5838 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:34:24.057669 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057615    5838 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:34:24.057669 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057625    5838 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:34:24.057669 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.057634    5838 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:34:24.058442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.058424    5838 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:34:24.058442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.058441    5838 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:34:24.059081 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.059062    5838 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:34:24.062381 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062364    5838 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:34:24.062381 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062380    5838 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:34:24.062506 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062496    5838 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 17:34:24.062708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062697    5838 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:34:24.062859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062744    5838 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:34:24.062859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062840    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:34:24.062859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062853    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:34:24.062859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062859    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062864    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062870    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062876    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062882    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062888    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062895    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062901    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062929    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062938    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062964    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:34:24.062983 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.062970    5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:34:24.066597 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.066581    5838 server.go:1263] "Started kubelet"
Nov 03 17:34:24.066773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.066696    5838 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:34:24.066876 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.066684    5838 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:34:24.066951 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.066925    5838 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:34:24.067319 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.067283    5838 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:34:24.067347 ip-10-0-139-156 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:34:24.072055 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.069196    5838 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.073988    5838 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.074006    5838 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.074117    5838 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.074130    5838 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.074122    5838 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:12 +0000 UTC" deadline="2028-04-17 07:58:23.834680359 +0000 UTC"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.074143    5838 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12734h23m59.7605401s"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.074140    5838 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-156.us-west-1.compute.internal\" not found"
Nov 03 17:34:24.075771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.074248    5838 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:34:24.077096 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077040    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077098    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077113    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2852e1-7207-4d27-a78e-ab6a06a425c3" volumeName="kubernetes.io/empty-dir/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-textfile" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077126    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb54bc0-f60d-4745-bcfe-5978119c0bde" volumeName="kubernetes.io/empty-dir/dbb54bc0-f60d-4745-bcfe-5978119c0bde-tmp-dir" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077138    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb54bc0-f60d-4745-bcfe-5978119c0bde" volumeName="kubernetes.io/projected/dbb54bc0-f60d-4745-bcfe-5978119c0bde-kube-api-access-2dgmd" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077149    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb9dbee-5655-45fc-8d5c-a51cfa17ea7d" volumeName="kubernetes.io/empty-dir/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-tmp-dir" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077161    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c628272-7a16-4897-af12-59bcfedd74cb" volumeName="kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077172    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077188    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config" seLinuxMountContext=""
Nov 03 17:34:24.077203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077200    5838 factory.go:55] Registering systemd factory
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077219    5838 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077201    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d0a499-8879-4147-b92b-b81d727ddff3" volumeName="kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077277    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077298    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2852e1-7207-4d27-a78e-ab6a06a425c3" volumeName="kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-tls" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077306    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb9dbee-5655-45fc-8d5c-a51cfa17ea7d" volumeName="kubernetes.io/configmap/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-config-volume" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077314    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb9dbee-5655-45fc-8d5c-a51cfa17ea7d" volumeName="kubernetes.io/projected/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-kube-api-access-59tph" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077326    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2021b15f-bca7-48eb-9280-bbbeb6c7bf9d" volumeName="kubernetes.io/projected/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kube-api-access-xq5j5" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077336    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077349    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b47b5da-225b-4e79-aab7-f81b12aa6191" volumeName="kubernetes.io/projected/4b47b5da-225b-4e79-aab7-f81b12aa6191-kube-api-access-r9bqd" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077357    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d0a499-8879-4147-b92b-b81d727ddff3" volumeName="kubernetes.io/configmap/c4d0a499-8879-4147-b92b-b81d727ddff3-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077365    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d0a499-8879-4147-b92b-b81d727ddff3" volumeName="kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077373    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6" volumeName="kubernetes.io/configmap/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077380    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6" volumeName="kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077391    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077400    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077408    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2852e1-7207-4d27-a78e-ab6a06a425c3" volumeName="kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077415    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6" volumeName="kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077423    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c628272-7a16-4897-af12-59bcfedd74cb" volumeName="kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077431    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-grpc-tls" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077440    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077452    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077459    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" volumeName="kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077467    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077475    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df9489d-1992-48e2-9fca-b6a83d7a0cff" volumeName="kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-tuned" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077483    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df9489d-1992-48e2-9fca-b6a83d7a0cff" volumeName="kubernetes.io/projected/1df9489d-1992-48e2-9fca-b6a83d7a0cff-kube-api-access-ddhm9" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077491    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077499    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077507    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d0a499-8879-4147-b92b-b81d727ddff3" volumeName="kubernetes.io/projected/c4d0a499-8879-4147-b92b-b81d727ddff3-kube-api-access-jwfjm" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077515    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2852e1-7207-4d27-a78e-ab6a06a425c3" volumeName="kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077522    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/configmap/25a774fe-c887-4a8c-96a6-ed3b863e344c-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077530    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077537    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077546    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" volumeName="kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077554    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" volumeName="kubernetes.io/projected/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-kube-api-access-jc4rf" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077571    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f48d878-d642-49c6-a4db-6e74ea2542fc" volumeName="kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-daemon-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077585    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e659c14-b9ea-4485-a903-eb9ed8e42272" volumeName="kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077592    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/empty-dir/25a774fe-c887-4a8c-96a6-ed3b863e344c-tmp" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077600    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077609    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config" seLinuxMountContext=""
Nov 03 17:34:24.077605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077622    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077631    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c628272-7a16-4897-af12-59bcfedd74cb" volumeName="kubernetes.io/projected/1c628272-7a16-4897-af12-59bcfedd74cb-kube-api-access-mgdz2" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077639    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077648    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" volumeName="kubernetes.io/secret/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077657    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="634768ef-d8f7-474d-9ee4-b479a1b1f17b" volumeName="kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077665    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e659c14-b9ea-4485-a903-eb9ed8e42272" volumeName="kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077674    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e659c14-b9ea-4485-a903-eb9ed8e42272" volumeName="kubernetes.io/projected/9e659c14-b9ea-4485-a903-eb9ed8e42272-kube-api-access-6nv6s" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077681    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb9dbee-5655-45fc-8d5c-a51cfa17ea7d" volumeName="kubernetes.io/secret/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-metrics-tls" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077690    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07057c45-a937-446c-96ea-c3e74244776b" volumeName="kubernetes.io/secret/07057c45-a937-446c-96ea-c3e74244776b-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077703    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c628272-7a16-4897-af12-59bcfedd74cb" volumeName="kubernetes.io/empty-dir/1c628272-7a16-4897-af12-59bcfedd74cb-volume-directive-shadow" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077710    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077719    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b47b5da-225b-4e79-aab7-f81b12aa6191" volumeName="kubernetes.io/configmap/4b47b5da-225b-4e79-aab7-f81b12aa6191-serviceca" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077747    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d4951c9-ab34-4727-9272-fc331a7fae42" volumeName="kubernetes.io/secret/7d4951c9-ab34-4727-9272-fc331a7fae42-agent-certs" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077756    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5739357-c9b8-445d-9b36-137395fd0857" volumeName="kubernetes.io/secret/c5739357-c9b8-445d-9b36-137395fd0857-tls-certificates" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077769    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077778    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6bc2a32-53d4-4ad7-a296-74dae0153ded" volumeName="kubernetes.io/projected/d6bc2a32-53d4-4ad7-a296-74dae0153ded-kube-api-access-rv8vr" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077792    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c628272-7a16-4897-af12-59bcfedd74cb" volumeName="kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077803    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077817    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077825    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077837    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" volumeName="kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-env-overrides" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077851    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f48d878-d642-49c6-a4db-6e74ea2542fc" volumeName="kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077862    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6" volumeName="kubernetes.io/projected/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-kube-api-access-xv6tv" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077874    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07057c45-a937-446c-96ea-c3e74244776b" volumeName="kubernetes.io/configmap/07057c45-a937-446c-96ea-c3e74244776b-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077886    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c628272-7a16-4897-af12-59bcfedd74cb" volumeName="kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077899    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df9489d-1992-48e2-9fca-b6a83d7a0cff" volumeName="kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-tmp" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077924    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077937    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077950    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6bc2a32-53d4-4ad7-a296-74dae0153ded" volumeName="kubernetes.io/secret/d6bc2a32-53d4-4ad7-a296-74dae0153ded-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077962    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0481c4-d60c-47f9-9fc5-901a405928bc" volumeName="kubernetes.io/projected/ea0481c4-d60c-47f9-9fc5-901a405928bc-kube-api-access-wprfg" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077974    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/projected/25a774fe-c887-4a8c-96a6-ed3b863e344c-kube-api-access-kqzwx" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077984    5838 factory.go:153] Registering CRI-O factory
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078003    5838 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078058    5838 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078079    5838 factory.go:103] Registering Raw factory
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078096    5838 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.077986    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3044c8b1-0ded-48b6-8496-5efad747ee0a" volumeName="kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078140    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078156    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6544fcbf-d2a3-48fc-8f79-71cc649dd82b" volumeName="kubernetes.io/projected/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-kube-api-access-jlwgl" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078170    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2260c22-12fe-411e-9a84-e8cd21d7389d" volumeName="kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078191    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078203    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078217    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46723e8e-0ca8-4209-9904-b35885a2f90f" volumeName="kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078229    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="634768ef-d8f7-474d-9ee4-b479a1b1f17b" volumeName="kubernetes.io/projected/634768ef-d8f7-474d-9ee4-b479a1b1f17b-kube-api-access-btmj5" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078243    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2852e1-7207-4d27-a78e-ab6a06a425c3" volumeName="kubernetes.io/projected/cb2852e1-7207-4d27-a78e-ab6a06a425c3-kube-api-access-snwrp" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078256    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0481c4-d60c-47f9-9fc5-901a405928bc" volumeName="kubernetes.io/secret/ea0481c4-d60c-47f9-9fc5-901a405928bc-cert" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078268    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a774fe-c887-4a8c-96a6-ed3b863e344c" volumeName="kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078279    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" volumeName="kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078289    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6544fcbf-d2a3-48fc-8f79-71cc649dd82b" volumeName="kubernetes.io/configmap/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-iptables-alerter-script" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078309    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d4951c9-ab34-4727-9272-fc331a7fae42" volumeName="kubernetes.io/configmap/7d4951c9-ab34-4727-9272-fc331a7fae42-konnectivity-ca" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078320    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f48d878-d642-49c6-a4db-6e74ea2542fc" volumeName="kubernetes.io/projected/7f48d878-d642-49c6-a4db-6e74ea2542fc-kube-api-access-8d4p6" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078331    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e659c14-b9ea-4485-a903-eb9ed8e42272" volumeName="kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078343    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f705b7-0fa2-44b0-a491-479876d4282a" volumeName="kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078354    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2852e1-7207-4d27-a78e-ab6a06a425c3" volumeName="kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078366    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07057c45-a937-446c-96ea-c3e74244776b" volumeName="kubernetes.io/empty-dir/07057c45-a937-446c-96ea-c3e74244776b-data-volume" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078376    5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07057c45-a937-446c-96ea-c3e74244776b" volumeName="kubernetes.io/projected/07057c45-a937-446c-96ea-c3e74244776b-kube-api-access-62slh" seLinuxMountContext=""
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078386    5838 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078394    5838 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:34:24.079681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.078832    5838 manager.go:319] Starting recovery of all containers
Nov 03 17:34:24.106645 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.106608    5838 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:34:24.108199 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.108170    5838 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:34:24.108199 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.108195    5838 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:34:24.108402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.108218    5838 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:34:24.108402 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.108266    5838 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 17:34:24.143627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.143437    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:34:24.147593 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.147570    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:34:24.150974 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.150942    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0" exitCode=0
Nov 03 17:34:24.157085 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.157051    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="19a07305db1c2e2061ff5c3dd2d7038a2fcbd171f617946d40e9d453be582db5" exitCode=0
Nov 03 17:34:24.157085 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.157085    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="9894e6736069653b1ed9583a9c254c2f62e89498bb138dc5cdb22e2cec4b72a2" exitCode=0
Nov 03 17:34:24.157309 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.157096    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="4a39e8e55dfeb0758af2d78cbb8715f8f9bef5f7e8ce2bbb0cc8f10b892db2fe" exitCode=0
Nov 03 17:34:24.157309 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.157106    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="12fd936c1541d8c9e9aaa9745a9eae0d492bf715fabb1fe95d8c763041f787e2" exitCode=0
Nov 03 17:34:24.157309 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.157114    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="67698d4652b85b5f7b2ef7d87b93f8cec3f4aacc9074280a17e9d9d649858097" exitCode=0
Nov 03 17:34:24.157309 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.157121    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e659c14-b9ea-4485-a903-eb9ed8e42272" containerID="810cd0824d66ec1eafdc0d9c1d54f0cf55d3edc8620a04eca17374a4110e870e" exitCode=0
Nov 03 17:34:24.161127 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.161104    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:34:24.161478 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.161447    5838 generic.go:356] "Generic (PLEG): container finished" podID="5908cbf5-2e94-4a1c-9c81-f7adfd472da0" containerID="74b847545ab15ba508effd70e706e9d242671b3f97b9fa3a6e5a87b601a185ce" exitCode=1
Nov 03 17:34:24.164763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.164720    5838 generic.go:356] "Generic (PLEG): container finished" podID="00fb4e1599f1bd3962469eeb7b079336" containerID="0b1cc32e422080cdb1c7ec14f3ce7132ca668607d8b10791f5bf2509efef69d6" exitCode=0
Nov 03 17:34:24.174219 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.174198    5838 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-156.us-west-1.compute.internal\" not found"
Nov 03 17:34:24.179370 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.179334    5838 generic.go:356] "Generic (PLEG): container finished" podID="cb2852e1-7207-4d27-a78e-ab6a06a425c3" containerID="9ebf5a77f7d8a0daae06d4f55be5a85028f8d58837a8f8f57d46d325e949c10a" exitCode=0
Nov 03 17:34:24.208345 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.208307    5838 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 17:34:24.208520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.208504    5838 manager.go:324] Recovery completed
Nov 03 17:34:24.231589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.231560    5838 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:34:24.233464 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.233444    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:34:24.233533 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.233479    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:34:24.233533 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.233490    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:34:24.236720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.236699    5838 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:34:24.236720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.236710    5838 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:34:24.236720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.236717    5838 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:34:24.236907 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.236790    5838 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:34:24.236972 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.236954    5838 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 17:34:24.237013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.236974    5838 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 17:34:24.237013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237001    5838 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 17:34:24.237013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237006    5838 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 17:34:24.237013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237012    5838 policy_none.go:50] "Start"
Nov 03 17:34:24.237118 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237023    5838 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:34:24.237118 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237035    5838 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:34:24.237183 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237173    5838 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 17:34:24.237213 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237185    5838 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 17:34:24.237213 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237190    5838 policy_none.go:44] "Start"
Nov 03 17:34:24.237882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237865    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:34:24.237928 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237898    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:34:24.237928 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.237908    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:34:24.242378 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.242351    5838 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:34:24.245158 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245145    5838 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:34:24.245235 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.245171    5838 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:34:24.245235 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245180    5838 server.go:87] "Starting device plugin registration server"
Nov 03 17:34:24.245499 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245486    5838 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:34:24.245564 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245500    5838 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:34:24.245624 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245599    5838 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:34:24.245796 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245756    5838 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:34:24.245858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245806    5838 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:34:24.245858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245813    5838 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:34:24.246043 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.245868    5838 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:34:24.245794372Z","UUID":"6b97b01a-227d-4866-8cdb-482286b8fbe5","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:34:24.247320 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.247302    5838 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:34:24.247431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.247329    5838 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:34:24.247725 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.247695    5838 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:34:24.247839 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.247775    5838 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-156.us-west-1.compute.internal\" not found"
Nov 03 17:34:24.274403 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.274314    5838 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:34:24.274403 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.274355    5838 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:34:24.274403 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.274396    5838 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 17:34:24.275223 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.275206    5838 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:34:24.345988 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.345953    5838 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:34:24.347232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.347210    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:34:24.347347 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.347249    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:34:24.347347 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.347260    5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-156.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:34:24.347347 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.347304    5838 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.373352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.373309    5838 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:34:24.408858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.408783    5838 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"]
Nov 03 17:34:24.409610 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409554    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"00fb4e1599f1bd3962469eeb7b079336","Type":"ContainerStarted","Data":"322e95c95a2335f1663ebc00932a5cb4c4e7e7593a7eed04030ff8a5dd7f915a"}
Nov 03 17:34:24.409610 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409597    5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.409766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409616    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"00fb4e1599f1bd3962469eeb7b079336","Type":"ContainerDied","Data":"0b1cc32e422080cdb1c7ec14f3ce7132ca668607d8b10791f5bf2509efef69d6"}
Nov 03 17:34:24.409766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409639    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"00fb4e1599f1bd3962469eeb7b079336","Type":"ContainerStarted","Data":"2918e9c082fecd5e3159436894e34a0da455017e06bfc5f41d649247a36b14c1"}
Nov 03 17:34:24.409766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409661    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"779eab1b0b29ccc85d3654a5ae82e6cd","Type":"ContainerStarted","Data":"216f471864ddf9f1d51998626b09dd4ac94741bebe637244e21ec5405353eab4"}
Nov 03 17:34:24.409766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409681    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal" event={"ID":"779eab1b0b29ccc85d3654a5ae82e6cd","Type":"ContainerStarted","Data":"f37cce55af08b05f56d85b2cced096147f14167a6f90ff346ca4af323d50d634"}
Nov 03 17:34:24.409766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.409663    5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.464127 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.464096    5838 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.464267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.464181    5838 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.476111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.476079    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.476283 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.476112    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.476283 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.476141    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/779eab1b0b29ccc85d3654a5ae82e6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"779eab1b0b29ccc85d3654a5ae82e6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.477334 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.477313    5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:34:24.477393 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.477380    5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.479017 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.479004    5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:34:24.479065 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:24.479041    5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.576584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.576487    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.576584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.576527    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/779eab1b0b29ccc85d3654a5ae82e6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"779eab1b0b29ccc85d3654a5ae82e6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.576584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.576545    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.576584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.576589    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.576867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.576597    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/779eab1b0b29ccc85d3654a5ae82e6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"779eab1b0b29ccc85d3654a5ae82e6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:24.576867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:24.576597    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fb4e1599f1bd3962469eeb7b079336-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" (UID: \"00fb4e1599f1bd3962469eeb7b079336\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:25.058830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.058796    5838 apiserver.go:52] "Watching apiserver"
Nov 03 17:34:25.196012 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.195980    5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:25.196202 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.196134    5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:25.215482 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.215455    5838 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:34:25.216161 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.216137    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-bs2m4","openshift-image-registry/node-ca-kpf92","openshift-monitoring/node-exporter-9znfw","kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal","openshift-dns/dns-default-rtvjp","openshift-network-diagnostics/network-check-target-tvv9c","openshift-network-operator/iptables-alerter-sdjkq","openshift-ovn-kubernetes/ovnkube-node-hwxhd","kube-system/global-pull-secret-syncer-cz52j","openshift-multus/network-metrics-daemon-8vqsh","openshift-monitoring/alertmanager-main-0","openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz","openshift-multus/multus-rxbv7","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7","openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr","openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd","openshift-cluster-node-tuning-operator/tuned-7xt6l","openshift-multus/multus-additional-cni-plugins-4tnrc","openshift-insights/insights-runtime-extractor-n7qdx","kube-system/konnectivity-agent-hxznn","openshift-console/console-5868466dd4-phngg","openshift-console/console-5f7fc76775-qdhqf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-lnx9n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v","openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q","openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"]
Nov 03 17:34:25.258624 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.258588    5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:34:25.258830 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:25.258653    5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:25.260061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.260044    5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:34:25.260112 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:25.260083    5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.us-west-1.compute.internal"
Nov 03 17:34:25.271583 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.271540    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:34:25.271717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.271591    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.271938 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.271912    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.272040 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.271979    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:34:25.272208 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272091    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.272208 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272109    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:34:25.272208 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272172    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.272311 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272244    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.272311 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272174    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.272311 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272274    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:34:25.272450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272375    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.272450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272383    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.272450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272404    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:34:25.272450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272409    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:34:25.272450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272413    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.272615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272508    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.272615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272554    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:34:25.272702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272634    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xhj6\""
Nov 03 17:34:25.272702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272666    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xxxqk\""
Nov 03 17:34:25.272702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272687    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\""
Nov 03 17:34:25.272809 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272712    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nx9c7\""
Nov 03 17:34:25.272809 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272752    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2vfjt\""
Nov 03 17:34:25.272809 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272759    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:34:25.272809 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272779    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:34:25.272809 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272687    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\""
Nov 03 17:34:25.273021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272814    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:34:25.273021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272817    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfxsl\""
Nov 03 17:34:25.273021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272818    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:34:25.273021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.272878    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:34:25.273021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273019    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.273247 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273084    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\""
Nov 03 17:34:25.273365 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273353    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.273432 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273418    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:34:25.273475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273447    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jxmj\""
Nov 03 17:34:25.273557 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273544    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.273599 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273544    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.273820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.273807    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qfzl\""
Nov 03 17:34:25.274349 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.274334    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.274633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.274619    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.274708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.274698    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tvvh\""
Nov 03 17:34:25.274904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.274891    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:34:25.274945 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.274924    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxb7s\""
Nov 03 17:34:25.275131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.275117    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:34:25.275633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.275621    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstfb\""
Nov 03 17:34:25.276493 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.276478    5838 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:34:25.279937 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.279918    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a774fe-c887-4a8c-96a6-ed3b863e344c-tmp\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.279989 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.279950    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.279989 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.279971    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-kubelet\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.280074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.279986    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.280074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280009    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.280074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280061    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.280205 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280163    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b47b5da-225b-4e79-aab7-f81b12aa6191-host\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.280205 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280177    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a774fe-c887-4a8c-96a6-ed3b863e344c-tmp\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.280302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280207    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-sys\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.280302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280181    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.280302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280240    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4p6\" (UniqueName: \"kubernetes.io/projected/7f48d878-d642-49c6-a4db-6e74ea2542fc-kube-api-access-8d4p6\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.280302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280261    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovn-node-metrics-cert\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.280302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280286    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.280494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280361    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.280494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280386    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-root\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.280494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280410    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-registration-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.280494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280432    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-modprobe-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.280494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280478    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-hostroot\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.280667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280511    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-tmp-dir\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.280667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280584    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-conf\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.280667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280596    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-tmp-dir\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.280667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280620    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-systemd\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.280667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280642    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-netns\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.280867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280672    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07057c45-a937-446c-96ea-c3e74244776b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.280867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280710    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.280867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280754    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovn-node-metrics-cert\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.280867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280791    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.280867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280838    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5739357-c9b8-445d-9b36-137395fd0857-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tb5x7\" (UID: \"c5739357-c9b8-445d-9b36-137395fd0857\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:25.280867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280865    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.281114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280883    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-cnibin\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.281114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280900    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.281114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.280990    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-ovn\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281087    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.281260 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281129    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.281260 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281152    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-grpc-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.281260 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281135    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.281260 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281179    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-socket-dir-parent\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.281260 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281197    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-script-lib\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281295    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.281420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281323    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfjm\" (UniqueName: \"kubernetes.io/projected/c4d0a499-8879-4147-b92b-b81d727ddff3-kube-api-access-jwfjm\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.281420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281353    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-etc-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281403    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-log-socket\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281403    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-script-lib\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281433    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4rf\" (UniqueName: \"kubernetes.io/projected/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-kube-api-access-jc4rf\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281460    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-wtmp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.281589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281568    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-netd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281590    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62slh\" (UniqueName: \"kubernetes.io/projected/07057c45-a937-446c-96ea-c3e74244776b-kube-api-access-62slh\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281620    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281657    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281676    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281696    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281726    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.281786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281766    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-kubernetes\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281799    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-k8s-cni-cncf-io\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281839    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281872    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-socket-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281898    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-tuned\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281924    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281960    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bqd\" (UniqueName: \"kubernetes.io/projected/4b47b5da-225b-4e79-aab7-f81b12aa6191-kube-api-access-r9bqd\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281964    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46723e8e-0ca8-4209-9904-b35885a2f90f-original-pull-secret\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.281984    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-device-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282016    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-tuned\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.282063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282051    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282086    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-bin\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282118    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5j5\" (UniqueName: \"kubernetes.io/projected/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kube-api-access-xq5j5\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282148    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb54bc0-f60d-4745-bcfe-5978119c0bde-hosts-file\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282192    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-cni-binary-copy\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282233    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282264    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a499-8879-4147-b92b-b81d727ddff3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282291    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprfg\" (UniqueName: \"kubernetes.io/projected/ea0481c4-d60c-47f9-9fc5-901a405928bc-kube-api-access-wprfg\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282322    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-tls\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282343    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282359    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-cni-binary-copy\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282360    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-kubelet\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282402    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-conf-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282424    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.282486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282446    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282510    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdz2\" (UniqueName: \"kubernetes.io/projected/1c628272-7a16-4897-af12-59bcfedd74cb-kube-api-access-mgdz2\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282525    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634768ef-d8f7-474d-9ee4-b479a1b1f17b-metrics-certs\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282618    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-var-lib-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282649    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysconfig\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282676    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282712    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv6s\" (UniqueName: \"kubernetes.io/projected/9e659c14-b9ea-4485-a903-eb9ed8e42272-kube-api-access-6nv6s\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282756    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b47b5da-225b-4e79-aab7-f81b12aa6191-serviceca\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282788    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282820    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282844    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-multus\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282885    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282891    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-system-cni-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.282962    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-node-log\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283005    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283032    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283071    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.283086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283092    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-var-lib-kubelet\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283107    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-bin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283124    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283205    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283224    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-dbus\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283238    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-sys\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283279    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwrp\" (UniqueName: \"kubernetes.io/projected/cb2852e1-7207-4d27-a78e-ab6a06a425c3-kube-api-access-snwrp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283310    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzwx\" (UniqueName: \"kubernetes.io/projected/25a774fe-c887-4a8c-96a6-ed3b863e344c-kube-api-access-kqzwx\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283349    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283376    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283400    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-cnibin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283428    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283466    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-sys-fs\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283495    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283527    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-system-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283553    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4951c9-ab34-4727-9272-fc331a7fae42-konnectivity-ca\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283561    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b47b5da-225b-4e79-aab7-f81b12aa6191-serviceca\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283578    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283604    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-slash\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283642    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-systemd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283651    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283659    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283694    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4951c9-ab34-4727-9272-fc331a7fae42-konnectivity-ca\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283696    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a774fe-c887-4a8c-96a6-ed3b863e344c-metrics-client-ca\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.283832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283826    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bc2a32-53d4-4ad7-a296-74dae0153ded-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283858    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-iptables-alerter-script\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283899    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-metrics-client-ca\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283926    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283956    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.283983    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284017    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284044    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-iptables-alerter-script\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284051    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07057c45-a937-446c-96ea-c3e74244776b-data-volume\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284083    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c628272-7a16-4897-af12-59bcfedd74cb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284102    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07057c45-a937-446c-96ea-c3e74244776b-data-volume\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284145    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c628272-7a16-4897-af12-59bcfedd74cb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284169    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284205    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/634768ef-d8f7-474d-9ee4-b479a1b1f17b-kube-api-access-btmj5\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284334    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-netns\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284370    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07057c45-a937-446c-96ea-c3e74244776b-crio-socket\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284399    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284428    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tph\" (UniqueName: \"kubernetes.io/projected/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-kube-api-access-59tph\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284451    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284474    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284502    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-textfile\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284542    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284560    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-textfile\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284576    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-host-slash\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284600    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-run\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284628    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284677    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284711    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0481c4-d60c-47f9-9fc5-901a405928bc-cert\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284812    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-lib-modules\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284855    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.284869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284865    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-tmp\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284890    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284939    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb54bc0-f60d-4745-bcfe-5978119c0bde-tmp-dir\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284960    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df9489d-1992-48e2-9fca-b6a83d7a0cff-tmp\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.284967    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285025    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb54bc0-f60d-4745-bcfe-5978119c0bde-tmp-dir\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285047    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4951c9-ab34-4727-9272-fc331a7fae42-agent-certs\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285082    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-metrics-tls\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285111    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8vr\" (UniqueName: \"kubernetes.io/projected/d6bc2a32-53d4-4ad7-a296-74dae0153ded-kube-api-access-rv8vr\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285155    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgmd\" (UniqueName: \"kubernetes.io/projected/dbb54bc0-f60d-4745-bcfe-5978119c0bde-kube-api-access-2dgmd\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285183    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-daemon-config\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285211    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-systemd-units\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285242    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-env-overrides\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285273    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285302    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285306    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4951c9-ab34-4727-9272-fc331a7fae42-agent-certs\") pod \"konnectivity-agent-hxznn\" (UID: \"7d4951c9-ab34-4727-9272-fc331a7fae42\") " pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285327    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285353    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-kubelet-config\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285384    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285387    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-daemon-config\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285389    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285447    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-env-overrides\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285479    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-host\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285511    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-etc-kubernetes\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285547    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6tv\" (UniqueName: \"kubernetes.io/projected/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-kube-api-access-xv6tv\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285567    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285582    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-config-volume\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285600    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwgl\" (UniqueName: \"kubernetes.io/projected/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-kube-api-access-jlwgl\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285616    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhm9\" (UniqueName: \"kubernetes.io/projected/1df9489d-1992-48e2-9fca-b6a83d7a0cff-kube-api-access-ddhm9\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285635    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-multus-certs\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285656    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285691    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-config\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285759    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07057c45-a937-446c-96ea-c3e74244776b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285878    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e659c14-b9ea-4485-a903-eb9ed8e42272-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.285994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.285902    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-ovnkube-config\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.299906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.299875    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:34:25.300051 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.299908    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.300051 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.299908    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.300226 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300174    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.300271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300228    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:34:25.300348 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300333    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:34:25.300431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300414    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v8zr6\""
Nov 03 17:34:25.300516 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300418    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:34:25.300614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300591    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.300751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300604    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.300751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300636    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.300751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300640    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:34:25.300751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300650    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jj5j\""
Nov 03 17:34:25.300751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300683    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6mmfj\""
Nov 03 17:34:25.300968 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300933    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 17:34:25.300968 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300959    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:34:25.301052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.300965    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 17:34:25.301052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.301010    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07057c45-a937-446c-96ea-c3e74244776b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.301117 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.301069    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.302237 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.302220    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.303449 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.303427    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.303496 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.303459    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.303496 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.303478    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a499-8879-4147-b92b-b81d727ddff3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.303570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.303558    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.304579 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.304567    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a774fe-c887-4a8c-96a6-ed3b863e344c-metrics-client-ca\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.304622 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.304608    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-metrics-client-ca\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.305763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.305746    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.305827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.305774    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-metrics-tls\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.305827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.305803    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0481c4-d60c-47f9-9fc5-901a405928bc-cert\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:25.305925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.305859    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.305925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.305903    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-config-volume\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.305997 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.305987    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07057c45-a937-446c-96ea-c3e74244776b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.309120 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.309048    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nctqr\""
Nov 03 17:34:25.309120 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.309057    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 17:34:25.309223 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.309137    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pwfx\""
Nov 03 17:34:25.309223 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.309160    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 17:34:25.310605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.310586    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.310680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.310641    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 17:34:25.310680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.310651    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 17:34:25.310791 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.310726    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 17:34:25.311870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.311852    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.311918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.311856    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.311918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.311858    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.311918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.311890    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.314150 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.314136    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.314269 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.314257    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.315273 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.315260    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4d0a499-8879-4147-b92b-b81d727ddff3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.320774 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.320755    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.320918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.320880    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 17:34:25.320995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.320938    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b97jd\""
Nov 03 17:34:25.321065 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.321050    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 17:34:25.322098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322069    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mvxtn\""
Nov 03 17:34:25.322187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322172    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:34:25.322187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322181    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:34:25.322315 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322248    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-58h0jjau7t32b\""
Nov 03 17:34:25.322315 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322250    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8sjq\""
Nov 03 17:34:25.322315 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322271    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 17:34:25.322454 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322338    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:34:25.322538 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322521    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b2s8h\""
Nov 03 17:34:25.322695 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322671    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:34:25.322767 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.322705    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 17:34:25.323126 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323103    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.323223 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323127    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.323223 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323185    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 17:34:25.323313 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323276    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.323592 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323567    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.323592 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323586    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.323788 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323770    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 17:34:25.323933 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323908    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 17:34:25.323995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.323949    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.324053 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.324001    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 17:34:25.324053 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.324030    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 17:34:25.324296 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.324279    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 17:34:25.324608 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.324593    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7lr2n\""
Nov 03 17:34:25.324938 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.324915    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c628272-7a16-4897-af12-59bcfedd74cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.325339 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.325322    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xlwcw\""
Nov 03 17:34:25.325449 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.325428    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.325492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.325433    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:34:25.325492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.325432    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 17:34:25.325699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.325686    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 17:34:25.326129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.326094    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 17:34:25.326307 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.326288    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 17:34:25.326383 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.326303    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 17:34:25.326383 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.326301    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 17:34:25.330433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.330417    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.331122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.330829    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.331122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.331034    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.331264 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.331237    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5739357-c9b8-445d-9b36-137395fd0857-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tb5x7\" (UID: \"c5739357-c9b8-445d-9b36-137395fd0857\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:25.331511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.331489    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-grpc-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.331671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.331653    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.334884 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.334812    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-tls\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.334975 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.334918    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.335346 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335322    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.335346 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335334    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.335453 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335374    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.335453 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335417    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.335453 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335435    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-tls\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.335610 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335488    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.335661 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.335635    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.350338 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.350309    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:34:25.350338 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.350329    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 17:34:25.350485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.350330    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 17:34:25.350485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.350316    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 17:34:25.350598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.350498    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351281    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4p6\" (UniqueName: \"kubernetes.io/projected/7f48d878-d642-49c6-a4db-6e74ea2542fc-kube-api-access-8d4p6\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351304    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4rf\" (UniqueName: \"kubernetes.io/projected/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-kube-api-access-jc4rf\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351451    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") pod \"console-5f7fc76775-qdhqf\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") " pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351483    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv6s\" (UniqueName: \"kubernetes.io/projected/9e659c14-b9ea-4485-a903-eb9ed8e42272-kube-api-access-6nv6s\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351593    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdz2\" (UniqueName: \"kubernetes.io/projected/1c628272-7a16-4897-af12-59bcfedd74cb-kube-api-access-mgdz2\") pod \"kube-state-metrics-5dc8d7d89b-sr54q\" (UID: \"1c628272-7a16-4897-af12-59bcfedd74cb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr54q"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351619    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5j5\" (UniqueName: \"kubernetes.io/projected/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kube-api-access-xq5j5\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351708    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") pod \"console-5868466dd4-phngg\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") " pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.351875    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62slh\" (UniqueName: \"kubernetes.io/projected/07057c45-a937-446c-96ea-c3e74244776b-kube-api-access-62slh\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352042    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfjm\" (UniqueName: \"kubernetes.io/projected/c4d0a499-8879-4147-b92b-b81d727ddff3-kube-api-access-jwfjm\") pod \"prometheus-operator-9cd89b74c-q2dlw\" (UID: \"c4d0a499-8879-4147-b92b-b81d727ddff3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q2dlw"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352050    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhm9\" (UniqueName: \"kubernetes.io/projected/1df9489d-1992-48e2-9fca-b6a83d7a0cff-kube-api-access-ddhm9\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352138    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bqd\" (UniqueName: \"kubernetes.io/projected/4b47b5da-225b-4e79-aab7-f81b12aa6191-kube-api-access-r9bqd\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352210    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/634768ef-d8f7-474d-9ee4-b479a1b1f17b-kube-api-access-btmj5\") pod \"network-metrics-daemon-8vqsh\" (UID: \"634768ef-d8f7-474d-9ee4-b479a1b1f17b\") " pod="openshift-multus/network-metrics-daemon-8vqsh"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352422    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwrp\" (UniqueName: \"kubernetes.io/projected/cb2852e1-7207-4d27-a78e-ab6a06a425c3-kube-api-access-snwrp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352436    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzwx\" (UniqueName: \"kubernetes.io/projected/25a774fe-c887-4a8c-96a6-ed3b863e344c-kube-api-access-kqzwx\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352518    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzb4\" (UniqueName: \"kubernetes.io/projected/a2260c22-12fe-411e-9a84-e8cd21d7389d-kube-api-access-zkzb4\") pod \"network-check-target-tvv9c\" (UID: \"a2260c22-12fe-411e-9a84-e8cd21d7389d\") " pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352526    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tph\" (UniqueName: \"kubernetes.io/projected/deb9dbee-5655-45fc-8d5c-a51cfa17ea7d-kube-api-access-59tph\") pod \"dns-default-rtvjp\" (UID: \"deb9dbee-5655-45fc-8d5c-a51cfa17ea7d\") " pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:25.352541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352516    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6tv\" (UniqueName: \"kubernetes.io/projected/e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6-kube-api-access-xv6tv\") pod \"openshift-state-metrics-7bd7489957-6s4qr\" (UID: \"e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6s4qr"
Nov 03 17:34:25.353079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352691    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgmd\" (UniqueName: \"kubernetes.io/projected/dbb54bc0-f60d-4745-bcfe-5978119c0bde-kube-api-access-2dgmd\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.353079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352878    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.353079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352916    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8vr\" (UniqueName: \"kubernetes.io/projected/d6bc2a32-53d4-4ad7-a296-74dae0153ded-kube-api-access-rv8vr\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:25.353079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.352968    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwgl\" (UniqueName: \"kubernetes.io/projected/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-kube-api-access-jlwgl\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.353187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.353101    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprfg\" (UniqueName: \"kubernetes.io/projected/ea0481c4-d60c-47f9-9fc5-901a405928bc-kube-api-access-wprfg\") pod \"ingress-canary-lnx9n\" (UID: \"ea0481c4-d60c-47f9-9fc5-901a405928bc\") " pod="openshift-ingress-canary/ingress-canary-lnx9n"
Nov 03 17:34:25.354828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.354809    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bc2a32-53d4-4ad7-a296-74dae0153ded-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ntxjz\" (UID: \"d6bc2a32-53d4-4ad7-a296-74dae0153ded\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:25.354922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.354861    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a774fe-c887-4a8c-96a6-ed3b863e344c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9d7c49b-8dzkd\" (UID: \"25a774fe-c887-4a8c-96a6-ed3b863e344c\") " pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:25.355946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.355932    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") pod \"alertmanager-main-0\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:34:25.386686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386654    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-conf\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.386686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386687    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-systemd\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386702    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-netns\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386719    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-cnibin\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386753    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-ovn\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386788    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-socket-dir-parent\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386812    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-systemd\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386824    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-cnibin\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386816    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-etc-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386844    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-ovn\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386812    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-netns\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386874    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-log-socket\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386874    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-conf\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386882    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-socket-dir-parent\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386912    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-etc-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.386922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386919    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-log-socket\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386927    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-wtmp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386961    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-node-exporter-wtmp\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386973    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-netd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.386999    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-netd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387001    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387035    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-kubernetes\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387059    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-k8s-cni-cncf-io\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387084    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387108    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-socket-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387120    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-k8s-cni-cncf-io\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387088    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysctl-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387111    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-kubernetes\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387137    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387137    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-device-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387179    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-device-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387180    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-socket-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387184    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387220    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-bin\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387237    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb54bc0-f60d-4745-bcfe-5978119c0bde-hosts-file\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387229    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387265    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-cni-bin\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387268    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387277    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb54bc0-f60d-4745-bcfe-5978119c0bde-hosts-file\") pod \"node-resolver-bs2m4\" (UID: \"dbb54bc0-f60d-4745-bcfe-5978119c0bde\") " pod="openshift-dns/node-resolver-bs2m4"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387298    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-kubelet\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387321    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-conf-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387334    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387350    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-var-lib-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387359    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-kubelet\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387367    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-conf-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387376    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysconfig\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387387    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-var-lib-openvswitch\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387405    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-multus\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387416    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-sysconfig\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387428    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-system-cni-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387444    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-node-log\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387459    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387476    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-system-cni-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387482    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-var-lib-kubelet\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387503    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-multus\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387506    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-bin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387511    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-node-log\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387531    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387541    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-var-lib-cni-bin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387538    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-var-lib-kubelet\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387549    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-dbus\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387582    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-dbus\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387583    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-sys\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.387596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387612    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-cnibin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387633    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-sys-fs\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387611    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-sys\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387661    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-system-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387695    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-slash\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387701    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-sys-fs\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387702    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-system-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387696    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-cnibin\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387718    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-systemd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387725    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-slash\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387763    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-run-systemd\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387764    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387787    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387804    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-netns\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387819    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07057c45-a937-446c-96ea-c3e74244776b-crio-socket\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387851    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-run-netns\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387852    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-host-slash\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387874    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6544fcbf-d2a3-48fc-8f79-71cc649dd82b-host-slash\") pod \"iptables-alerter-sdjkq\" (UID: \"6544fcbf-d2a3-48fc-8f79-71cc649dd82b\") " pod="openshift-network-operator/iptables-alerter-sdjkq"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387880    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-run\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387879    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387897    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-lib-modules\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387913    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387925    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07057c45-a937-446c-96ea-c3e74244776b-crio-socket\") pod \"insights-runtime-extractor-n7qdx\" (UID: \"07057c45-a937-446c-96ea-c3e74244776b\") " pod="openshift-insights/insights-runtime-extractor-n7qdx"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387934    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-systemd-units\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387937    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-multus-cni-dir\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387952    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-kubelet-config\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387966    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-host\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387974    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e659c14-b9ea-4485-a903-eb9ed8e42272-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tnrc\" (UID: \"9e659c14-b9ea-4485-a903-eb9ed8e42272\") " pod="openshift-multus/multus-additional-cni-plugins-4tnrc"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387981    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-systemd-units\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387933    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-run\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387988    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-etc-kubernetes\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.387999    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46723e8e-0ca8-4209-9904-b35885a2f90f-kubelet-config\") pod \"global-pull-secret-syncer-cz52j\" (UID: \"46723e8e-0ca8-4209-9904-b35885a2f90f\") " pod="kube-system/global-pull-secret-syncer-cz52j"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388018    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-etc-kubernetes\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388023    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-host\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388031    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-multus-certs\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388025    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-lib-modules\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388054    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-kubelet\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388075    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-host-run-multus-certs\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388088    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b47b5da-225b-4e79-aab7-f81b12aa6191-host\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388102    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-sys\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388118    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-root\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388131    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5908cbf5-2e94-4a1c-9c81-f7adfd472da0-host-kubelet\") pod \"ovnkube-node-hwxhd\" (UID: \"5908cbf5-2e94-4a1c-9c81-f7adfd472da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388135    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-registration-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388177    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-modprobe-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388181    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-sys\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388174    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb2852e1-7207-4d27-a78e-ab6a06a425c3-root\") pod \"node-exporter-9znfw\" (UID: \"cb2852e1-7207-4d27-a78e-ab6a06a425c3\") " pod="openshift-monitoring/node-exporter-9znfw"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388177    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2021b15f-bca7-48eb-9280-bbbeb6c7bf9d-registration-dir\") pod \"aws-ebs-csi-driver-node-bz27v\" (UID: \"2021b15f-bca7-48eb-9280-bbbeb6c7bf9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz27v"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388196    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b47b5da-225b-4e79-aab7-f81b12aa6191-host\") pod \"node-ca-kpf92\" (UID: \"4b47b5da-225b-4e79-aab7-f81b12aa6191\") " pod="openshift-image-registry/node-ca-kpf92"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388202    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-hostroot\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388232    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f48d878-d642-49c6-a4db-6e74ea2542fc-hostroot\") pod \"multus-rxbv7\" (UID: \"7f48d878-d642-49c6-a4db-6e74ea2542fc\") " pod="openshift-multus/multus-rxbv7"
Nov 03 17:34:25.389157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.388247    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1df9489d-1992-48e2-9fca-b6a83d7a0cff-etc-modprobe-d\") pod \"tuned-7xt6l\" (UID: \"1df9489d-1992-48e2-9fca-b6a83d7a0cff\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6l"
Nov 03 17:34:25.869781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.869751    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:25.964238 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:25.964202    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7fc76775-qdhqf"]
Nov 03 17:34:26.462380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:26.462341    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:26.467388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:26.467354    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:27.216633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.216601    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86d7f486f8-7vxp2"]
Nov 03 17:34:27.221835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.221814    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.282137 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.282097    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d7f486f8-7vxp2"]
Nov 03 17:34:27.306766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.306713    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phlq8\" (UniqueName: \"kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.306917 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.306862    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.306917 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.306910    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.307018 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.307003    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.307277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.307252    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.307402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.307374    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.307474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.307426    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.307474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.307445    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.408826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408785    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phlq8\" (UniqueName: \"kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.408826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408828    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408846    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408872    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408907    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408936    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408967    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.408984    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.409251    5838 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:34:27.409513 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.409495    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.409891 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.409863    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.410000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.409891    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.410000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.409897    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.410123 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.410103    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.411592 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.411573    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.411870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.411849    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.472905 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.472836    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phlq8\" (UniqueName: \"kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8\") pod \"console-86d7f486f8-7vxp2\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") " pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.521861 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.521830    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rtvjp"
Nov 03 17:34:27.533399 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.533373    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:27.558951 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.558920    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:27.962588 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:27.962557    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d7f486f8-7vxp2"]
Nov 03 17:34:28.208695 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:28.208652    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d7f486f8-7vxp2" event={"ID":"5f6bc412-5a3c-4826-a2df-123014d49371","Type":"ContainerStarted","Data":"fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3"}
Nov 03 17:34:28.208918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:28.208708    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d7f486f8-7vxp2" event={"ID":"5f6bc412-5a3c-4826-a2df-123014d49371","Type":"ContainerStarted","Data":"1d4415cfaa4b885d8e73e1ba8145e97ac787328baae7bbdcb215bcc5f99c6a92"}
Nov 03 17:34:28.331387 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:28.331251    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86d7f486f8-7vxp2" podStartSLOduration=1.331232414 podStartE2EDuration="1.331232414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:28.330434814 +0000 UTC m=+4.341172948" watchObservedRunningTime="2026-11-03 17:34:28.331232414 +0000 UTC m=+4.341970484"
Nov 03 17:34:28.687072 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:28.687041    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:29.078185 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.078101    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:29.209858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.209814    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:29.217488 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.217459    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:29.220816 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.220796    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ntxjz"
Nov 03 17:34:29.368052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.367960    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:29.455184 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.455152    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tb5x7"
Nov 03 17:34:29.630548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.630454    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:29.838914 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:29.838861    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:30.175623 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:30.175588    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:30.181298 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:30.181270    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85d9d7c49b-8dzkd"
Nov 03 17:34:30.290121 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:30.290082    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:30.480784 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:30.480673    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:31.974753 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:31.974703    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:32.145041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:32.144989    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:32.151308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:32.151283    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:32.151542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:32.151527    5838 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:34:33.402757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:33.402655    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:33.865959 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:33.865879    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:33.870822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:33.870794    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:34.234119 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:34.234092    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:34:34.408459 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:34.408427    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5868466dd4-phngg"]
Nov 03 17:34:35.766148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:35.766106    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hxznn"
Nov 03 17:34:36.181721 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:36.181687    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:36.346095 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:36.346056    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:38.016949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:38.016912    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:38.214261 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:38.214225    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:40.338283 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:40.338240    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:34:43.528691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:43.528651    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:43.706927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:43.706894    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:46.931229 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:46.931182    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:47.088741 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:47.088689    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:48.777475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:48.777443    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwxhd"
Nov 03 17:34:51.218948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.218888    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f7fc76775-qdhqf" podUID="3044c8b1-0ded-48b6-8496-5efad747ee0a" containerName="console" containerID="cri-o://144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af" gracePeriod=15
Nov 03 17:34:51.474510 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.474447    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:51.637370 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637329    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637392    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637409    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637431    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637465    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637481    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637550    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") pod \"3044c8b1-0ded-48b6-8496-5efad747ee0a\" (UID: \"3044c8b1-0ded-48b6-8496-5efad747ee0a\") "
Nov 03 17:34:51.637927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637883    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:34:51.638028 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637930    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:51.638074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.637933    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:51.638137 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.638111    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:51.640095 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.640068    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "kube-api-access-dcdbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:51.640350 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.640326    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:51.640429 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.640398    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config" pod "3044c8b1-0ded-48b6-8496-5efad747ee0a" (UID: "3044c8b1-0ded-48b6-8496-5efad747ee0a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738864    5838 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738894    5838 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-oauth-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738904    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcdbn\" (UniqueName: \"kubernetes.io/projected/3044c8b1-0ded-48b6-8496-5efad747ee0a-kube-api-access-dcdbn\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738916    5838 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3044c8b1-0ded-48b6-8496-5efad747ee0a-tmp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738926    5838 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-oauth-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738934    5838 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:51.738948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:51.738943    5838 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3044c8b1-0ded-48b6-8496-5efad747ee0a-console-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:52.297317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.297281    5838 generic.go:356] "Generic (PLEG): container finished" podID="3044c8b1-0ded-48b6-8496-5efad747ee0a" containerID="144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af" exitCode=0
Nov 03 17:34:52.297839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.297333    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7fc76775-qdhqf" event={"ID":"3044c8b1-0ded-48b6-8496-5efad747ee0a","Type":"ContainerDied","Data":"144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af"}
Nov 03 17:34:52.297839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.297336    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7fc76775-qdhqf"
Nov 03 17:34:52.297839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.297372    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7fc76775-qdhqf" event={"ID":"3044c8b1-0ded-48b6-8496-5efad747ee0a","Type":"ContainerDied","Data":"4bf1739aa502e67b380fc4993c6e480af42a7f0a4d3ac9e2127228773de7ecd2"}
Nov 03 17:34:52.297839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.297391    5838 scope.go:122] "RemoveContainer" containerID="144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af"
Nov 03 17:34:52.305523 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.305505    5838 scope.go:122] "RemoveContainer" containerID="144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af"
Nov 03 17:34:52.305827 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:34:52.305796    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af\": container with ID starting with 144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af not found: ID does not exist" containerID="144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af"
Nov 03 17:34:52.305947 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.305835    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af"} err="failed to get container status \"144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af\": rpc error: code = NotFound desc = could not find container \"144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af\": container with ID starting with 144e24c3e5cfa0fad68480d93ef27cb95766ab259168c79b33c275ebf0e9f2af not found: ID does not exist"
Nov 03 17:34:52.465354 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.465316    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7fc76775-qdhqf"]
Nov 03 17:34:52.515620 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:52.515586    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f7fc76775-qdhqf"]
Nov 03 17:34:53.419394 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:53.419359    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:53.571622 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:53.571591    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:54.113492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:54.113456    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3044c8b1-0ded-48b6-8496-5efad747ee0a" path="/var/lib/kubelet/pods/3044c8b1-0ded-48b6-8496-5efad747ee0a/volumes"
Nov 03 17:34:56.201828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:56.201796    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tvv9c"
Nov 03 17:34:58.312314 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:58.312279    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:58.472325 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:58.472288    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:58.636764 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:58.636647    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:59.428449 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.428398    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5868466dd4-phngg" podUID="c7f705b7-0fa2-44b0-a491-479876d4282a" containerName="console" containerID="cri-o://beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297" gracePeriod=15
Nov 03 17:34:59.707363 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.707293    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:34:59.802436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802400    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.802609 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802449    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.802609 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802521    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.802609 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802577    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.802609 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802604    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.802827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802635    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.802882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802843    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.803176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802975    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:34:59.803176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.802992    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:59.803176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.803068    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:59.803176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.803100    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:59.803176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.803107    5838 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7f705b7-0fa2-44b0-a491-479876d4282a-tmp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.803176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.803133    5838 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-oauth-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.804957 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.804931    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:59.804957 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.804937    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "kube-api-access-vq9dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:59.805212 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.805186    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:59.904001 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.903961    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") pod \"c7f705b7-0fa2-44b0-a491-479876d4282a\" (UID: \"c7f705b7-0fa2-44b0-a491-479876d4282a\") "
Nov 03 17:34:59.904201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.904149    5838 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.904201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.904163    5838 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-oauth-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.904201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.904173    5838 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f705b7-0fa2-44b0-a491-479876d4282a-console-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.904201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.904183    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9dm\" (UniqueName: \"kubernetes.io/projected/c7f705b7-0fa2-44b0-a491-479876d4282a-kube-api-access-vq9dm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.904201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.904195    5838 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-trusted-ca-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:34:59.904469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:34:59.904396    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config" pod "c7f705b7-0fa2-44b0-a491-479876d4282a" (UID: "c7f705b7-0fa2-44b0-a491-479876d4282a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:00.005058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.004973    5838 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7f705b7-0fa2-44b0-a491-479876d4282a-console-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:00.328165 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.328075    5838 generic.go:356] "Generic (PLEG): container finished" podID="c7f705b7-0fa2-44b0-a491-479876d4282a" containerID="beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297" exitCode=0
Nov 03 17:35:00.328165 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.328128    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5868466dd4-phngg"
Nov 03 17:35:00.328345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.328160    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5868466dd4-phngg" event={"ID":"c7f705b7-0fa2-44b0-a491-479876d4282a","Type":"ContainerDied","Data":"beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297"}
Nov 03 17:35:00.328345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.328227    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5868466dd4-phngg" event={"ID":"c7f705b7-0fa2-44b0-a491-479876d4282a","Type":"ContainerDied","Data":"c519d15b3ce0fb1534c197ce30c33277ac52a34767a23d8eb93765c2f4f98568"}
Nov 03 17:35:00.328345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.328248    5838 scope.go:122] "RemoveContainer" containerID="beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297"
Nov 03 17:35:00.337720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.337689    5838 scope.go:122] "RemoveContainer" containerID="beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297"
Nov 03 17:35:00.338045 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:00.338018    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297\": container with ID starting with beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297 not found: ID does not exist" containerID="beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297"
Nov 03 17:35:00.338099 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.338054    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297"} err="failed to get container status \"beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297\": rpc error: code = NotFound desc = could not find container \"beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297\": container with ID starting with beea03914435ede1952f86f0fac2526e8945ae4422068a009d11094d6cc72297 not found: ID does not exist"
Nov 03 17:35:00.497322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.497293    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5868466dd4-phngg"]
Nov 03 17:35:00.547543 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:00.547513    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5868466dd4-phngg"]
Nov 03 17:35:01.934779 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:01.934725    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:02.089958 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:02.089921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:02.113712 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:02.113677    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f705b7-0fa2-44b0-a491-479876d4282a" path="/var/lib/kubelet/pods/c7f705b7-0fa2-44b0-a491-479876d4282a/volumes"
Nov 03 17:35:05.859534 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:05.859499    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:06.018314 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:06.018279    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:08.071229 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071188    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:35:08.071649 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071503    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="alertmanager" containerID="cri-o://befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb" gracePeriod=120
Nov 03 17:35:08.071649 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071570    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy" containerID="cri-o://da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e" gracePeriod=120
Nov 03 17:35:08.071649 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071569    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-metric" containerID="cri-o://24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70" gracePeriod=120
Nov 03 17:35:08.071649 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071591    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="config-reloader" containerID="cri-o://ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352" gracePeriod=120
Nov 03 17:35:08.071899 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071653    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-web" containerID="cri-o://9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b" gracePeriod=120
Nov 03 17:35:08.072458 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.071930    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="prom-label-proxy" containerID="cri-o://d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd" gracePeriod=120
Nov 03 17:35:08.327285 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.327222    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.364002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.363971    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd" exitCode=0
Nov 03 17:35:08.364002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.363995    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70" exitCode=0
Nov 03 17:35:08.364002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364001    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e" exitCode=0
Nov 03 17:35:08.364002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364007    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b" exitCode=0
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364015    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352" exitCode=0
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364023    5838 generic.go:356] "Generic (PLEG): container finished" podID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb" exitCode=0
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364037    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"}
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364049    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364115    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"}
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364146    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"}
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364176    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"}
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364183    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364213    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"}
Nov 03 17:35:08.364267 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364255    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"}
Nov 03 17:35:08.364617 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.364293    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36c4adf8-f640-4c4e-b92b-2c5a7f901b11","Type":"ContainerDied","Data":"9b322cb8d668b170ed2b24a84323dbdd0cffcd4a4d6e8ae39209b39d1d65662d"}
Nov 03 17:35:08.372054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372034    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.372636 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372617    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372706 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372656    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372706 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372683    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372706 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372704    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372745    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372786    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372843    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.372889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372883    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.373142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372912    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.373142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372951    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.373142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.372976    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.373142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.373039    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.373142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.373069    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") pod \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\" (UID: \"36c4adf8-f640-4c4e-b92b-2c5a7f901b11\") "
Nov 03 17:35:08.373930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.373420    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:08.373930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.373603    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:35:08.375012 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.374965    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:08.376579 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.376546    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "kube-api-access-s4h5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:08.376579 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.376565    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.377726 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.377698    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.377904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.377796    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:35:08.377904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.377828    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.378226 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.378203    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:08.378313 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.378297    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.378891 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.378867    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.382032 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.382010    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.382032 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.382026    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.389117 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.389093    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config" pod "36c4adf8-f640-4c4e-b92b-2c5a7f901b11" (UID: "36c4adf8-f640-4c4e-b92b-2c5a7f901b11"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:08.389762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.389713    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.396461 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.396443    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.403470 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.403438    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.410667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.410649    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.417909 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.417893    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.418158 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.418136    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.418215 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418167    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"} err="failed to get container status \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist"
Nov 03 17:35:08.418215 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418185    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.418421 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.418405    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.418462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418427    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"} err="failed to get container status \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist"
Nov 03 17:35:08.418462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418441    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.418666 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.418649    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.418714 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418669    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"} err="failed to get container status \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist"
Nov 03 17:35:08.418714 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418681    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.418907 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.418892    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.418947 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418911    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"} err="failed to get container status \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist"
Nov 03 17:35:08.418947 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.418923    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.419150 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.419134    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.419198 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419153    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"} err="failed to get container status \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist"
Nov 03 17:35:08.419198 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419167    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.419344 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.419329    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.419382 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419346    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"} err="failed to get container status \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist"
Nov 03 17:35:08.419382 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419357    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.419584 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:08.419556    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.419630 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419587    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"} err="failed to get container status \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist"
Nov 03 17:35:08.419630 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419599    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.419835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419815    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"} err="failed to get container status \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist"
Nov 03 17:35:08.419889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.419837    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.420063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420046    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"} err="failed to get container status \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist"
Nov 03 17:35:08.420063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420063    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.420255 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420238    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"} err="failed to get container status \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist"
Nov 03 17:35:08.420295 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420255    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.420460 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420444    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"} err="failed to get container status \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist"
Nov 03 17:35:08.420460 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420459    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.420658 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420641    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"} err="failed to get container status \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist"
Nov 03 17:35:08.420658 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420656    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.420888 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420872    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"} err="failed to get container status \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist"
Nov 03 17:35:08.420888 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.420888    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.421104 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421087    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"} err="failed to get container status \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist"
Nov 03 17:35:08.421149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421104    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.421302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421287    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"} err="failed to get container status \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist"
Nov 03 17:35:08.421345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421301    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.421487 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421472    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"} err="failed to get container status \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist"
Nov 03 17:35:08.421528 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421486    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.421669 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421653    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"} err="failed to get container status \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist"
Nov 03 17:35:08.421708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421668    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.421883 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421859    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"} err="failed to get container status \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist"
Nov 03 17:35:08.421943 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.421883    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.422090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422069    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"} err="failed to get container status \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist"
Nov 03 17:35:08.422127 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422092    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.422321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422300    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"} err="failed to get container status \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist"
Nov 03 17:35:08.422321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422320    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.422577 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422557    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"} err="failed to get container status \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist"
Nov 03 17:35:08.422651 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422579    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.422811 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422794    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"} err="failed to get container status \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist"
Nov 03 17:35:08.422863 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422811    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.423013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.422995    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"} err="failed to get container status \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist"
Nov 03 17:35:08.423082 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423014    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.423228 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423211    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"} err="failed to get container status \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist"
Nov 03 17:35:08.423271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423228    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.423407 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423390    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"} err="failed to get container status \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist"
Nov 03 17:35:08.423474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423408    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.423612 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423595    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"} err="failed to get container status \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist"
Nov 03 17:35:08.423656 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423613    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.423821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423803    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"} err="failed to get container status \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist"
Nov 03 17:35:08.423896 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.423822    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.424072 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424055    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"} err="failed to get container status \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist"
Nov 03 17:35:08.424120 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424074    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.424300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424278    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"} err="failed to get container status \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist"
Nov 03 17:35:08.424348 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424300    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.424512 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424495    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"} err="failed to get container status \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist"
Nov 03 17:35:08.424581 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424514    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.424745 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424713    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"} err="failed to get container status \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist"
Nov 03 17:35:08.424796 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424755    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.424955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424940    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"} err="failed to get container status \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist"
Nov 03 17:35:08.424999 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.424955    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.425168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425149    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"} err="failed to get container status \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist"
Nov 03 17:35:08.425168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425164    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.425433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425412    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"} err="failed to get container status \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist"
Nov 03 17:35:08.425485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425435    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.425681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425663    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"} err="failed to get container status \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist"
Nov 03 17:35:08.425778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425681    5838 scope.go:122] "RemoveContainer" containerID="d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"
Nov 03 17:35:08.425929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425908    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd"} err="failed to get container status \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": rpc error: code = NotFound desc = could not find container \"d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd\": container with ID starting with d9f407687de41ee749185e6cb5fd800db9add347d4e62e12a8dd733ebdc3c9fd not found: ID does not exist"
Nov 03 17:35:08.425977 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.425929    5838 scope.go:122] "RemoveContainer" containerID="24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"
Nov 03 17:35:08.426142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426126    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70"} err="failed to get container status \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": rpc error: code = NotFound desc = could not find container \"24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70\": container with ID starting with 24adbe5d0141c4dc62881187937eadac6257483fc0f21cc5636d887ea26e2f70 not found: ID does not exist"
Nov 03 17:35:08.426187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426142    5838 scope.go:122] "RemoveContainer" containerID="da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"
Nov 03 17:35:08.426346 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426326    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e"} err="failed to get container status \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": rpc error: code = NotFound desc = could not find container \"da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e\": container with ID starting with da557aed5b5c9c63c6a1ba5c05b30bb0739626bcb43325dd2fbc7556667a474e not found: ID does not exist"
Nov 03 17:35:08.426385 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426348    5838 scope.go:122] "RemoveContainer" containerID="9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"
Nov 03 17:35:08.426541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426524    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b"} err="failed to get container status \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": rpc error: code = NotFound desc = could not find container \"9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b\": container with ID starting with 9e3160c7b8bdcd4b803da900add104cdab9a3cc79a48240444b47506127dc19b not found: ID does not exist"
Nov 03 17:35:08.426578 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426541    5838 scope.go:122] "RemoveContainer" containerID="ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"
Nov 03 17:35:08.426743 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426713    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352"} err="failed to get container status \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": rpc error: code = NotFound desc = could not find container \"ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352\": container with ID starting with ad12530be24fde34c19c417b9c72360355b1e0d99e1b103f83ef2ee043730352 not found: ID does not exist"
Nov 03 17:35:08.426792 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426746    5838 scope.go:122] "RemoveContainer" containerID="befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"
Nov 03 17:35:08.426969 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426950    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb"} err="failed to get container status \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": rpc error: code = NotFound desc = could not find container \"befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb\": container with ID starting with befb4daec7fcc55156ad5dbd6af33d61c6942dfa19c7dd3cda7c04757d1c28bb not found: ID does not exist"
Nov 03 17:35:08.427015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.426969    5838 scope.go:122] "RemoveContainer" containerID="cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"
Nov 03 17:35:08.427205 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.427190    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0"} err="failed to get container status \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": rpc error: code = NotFound desc = could not find container \"cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0\": container with ID starting with cff324e030c7683cdc2ab731040183e9fd4ac95c214784c986aa34a425c984f0 not found: ID does not exist"
Nov 03 17:35:08.474520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474488    5838 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-web-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474514    5838 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-tls-assets\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474526    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4h5z\" (UniqueName: \"kubernetes.io/projected/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-kube-api-access-s4h5z\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474536    5838 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-out\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474554    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-metrics-client-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474565    5838 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-main-db\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474575    5838 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474584    5838 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474593    5838 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474606    5838 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-main-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474620    5838 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474633    5838 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-cluster-tls-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.474920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.474643    5838 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36c4adf8-f640-4c4e-b92b-2c5a7f901b11-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:08.792060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.792023    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:35:08.797843 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.797818    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:35:08.833006 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.832976    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:35:08.833291 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833277    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-web"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833291    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-web"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833304    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="prom-label-proxy"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833309    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="prom-label-proxy"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833320    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3044c8b1-0ded-48b6-8496-5efad747ee0a" containerName="console"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833325    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3044c8b1-0ded-48b6-8496-5efad747ee0a" containerName="console"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833332    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="alertmanager"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833337    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="alertmanager"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833346    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833350    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833357    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="config-reloader"
Nov 03 17:35:08.833356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833361    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="config-reloader"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833367    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-metric"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833371    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-metric"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833379    5838 cpu_manager.go:414] "containerMap: removing container" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="init-config-reloader"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833383    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="init-config-reloader"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833390    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c7f705b7-0fa2-44b0-a491-479876d4282a" containerName="console"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833395    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f705b7-0fa2-44b0-a491-479876d4282a" containerName="console"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833441    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-web"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833449    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="prom-label-proxy"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833456    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3044c8b1-0ded-48b6-8496-5efad747ee0a" containerName="console"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833461    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="alertmanager"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833468    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833473    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="config-reloader"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833478    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="kube-rbac-proxy-metric"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833483    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" containerName="init-config-reloader"
Nov 03 17:35:08.833686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.833487    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f705b7-0fa2-44b0-a491-479876d4282a" containerName="console"
Nov 03 17:35:08.836989 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.836971    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877344    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b802a022-e188-4411-806c-d8ccb0e22450-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877546 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877389    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877546 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877408    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qr6\" (UniqueName: \"kubernetes.io/projected/b802a022-e188-4411-806c-d8ccb0e22450-kube-api-access-n4qr6\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877546 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877449    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b802a022-e188-4411-806c-d8ccb0e22450-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877538    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b802a022-e188-4411-806c-d8ccb0e22450-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877570    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877597    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877631    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b802a022-e188-4411-806c-d8ccb0e22450-config-out\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877645    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b802a022-e188-4411-806c-d8ccb0e22450-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877746    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877770    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877795    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-web-config\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.877827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.877812    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-config-volume\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.892638 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892595    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 17:35:08.892638 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892623    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 17:35:08.892894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892665    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 17:35:08.892894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892675    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 17:35:08.892894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892786    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 17:35:08.892894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892842    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b2s8h\""
Nov 03 17:35:08.893046 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892898    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 17:35:08.893046 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.892960    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 17:35:08.893111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.893092    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 17:35:08.893436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.893423    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 17:35:08.964899 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.964825    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:35:08.978240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978203    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qr6\" (UniqueName: \"kubernetes.io/projected/b802a022-e188-4411-806c-d8ccb0e22450-kube-api-access-n4qr6\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978245    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b802a022-e188-4411-806c-d8ccb0e22450-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978268    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b802a022-e188-4411-806c-d8ccb0e22450-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978285    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978411    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978452    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b802a022-e188-4411-806c-d8ccb0e22450-config-out\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978478    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b802a022-e188-4411-806c-d8ccb0e22450-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978527    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978561    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978606    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-web-config\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978637    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-config-volume\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978674    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b802a022-e188-4411-806c-d8ccb0e22450-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.978929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.978754    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.979782 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.979725    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b802a022-e188-4411-806c-d8ccb0e22450-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.979950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.979920    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b802a022-e188-4411-806c-d8ccb0e22450-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.980066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.980002    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b802a022-e188-4411-806c-d8ccb0e22450-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.981882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.981851    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.982176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.982145    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b802a022-e188-4411-806c-d8ccb0e22450-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.982514 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.982496    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.982653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.982630    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-config-volume\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.982720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.982633    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b802a022-e188-4411-806c-d8ccb0e22450-config-out\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.982720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.982701    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.982850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.982762    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.983195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.983178    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:08.983699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:08.983672    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b802a022-e188-4411-806c-d8ccb0e22450-web-config\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:09.046102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:09.046071    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qr6\" (UniqueName: \"kubernetes.io/projected/b802a022-e188-4411-806c-d8ccb0e22450-kube-api-access-n4qr6\") pod \"alertmanager-main-0\" (UID: \"b802a022-e188-4411-806c-d8ccb0e22450\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:09.154249 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:09.154212    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:35:09.589626 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:09.589600    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:35:09.591688 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:35:09.591657    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb802a022_e188_4411_806c_d8ccb0e22450.slice/crio-c7e21404add2eb606eff30d326a3eed1737caa62d0ef1ac800e9b45c7852dafd WatchSource:0}: Error finding container c7e21404add2eb606eff30d326a3eed1737caa62d0ef1ac800e9b45c7852dafd: Status 404 returned error can't find the container with id c7e21404add2eb606eff30d326a3eed1737caa62d0ef1ac800e9b45c7852dafd
Nov 03 17:35:10.114222 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:10.114174    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c4adf8-f640-4c4e-b92b-2c5a7f901b11" path="/var/lib/kubelet/pods/36c4adf8-f640-4c4e-b92b-2c5a7f901b11/volumes"
Nov 03 17:35:10.337052 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:10.337013    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:35:10.372992 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:10.372901    5838 generic.go:356] "Generic (PLEG): container finished" podID="b802a022-e188-4411-806c-d8ccb0e22450" containerID="2a16cf92875609bb82521c672eb3a6cb9ee753b868248094a7bcf90603815e1c" exitCode=0
Nov 03 17:35:10.372992 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:10.372980    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerDied","Data":"2a16cf92875609bb82521c672eb3a6cb9ee753b868248094a7bcf90603815e1c"}
Nov 03 17:35:10.373161 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:10.373073    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"c7e21404add2eb606eff30d326a3eed1737caa62d0ef1ac800e9b45c7852dafd"}
Nov 03 17:35:11.380064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.380030    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"a3c31181ddcdfc1caef6f47a11de6a35a76d1fd797b4764c9c36b0738456028d"}
Nov 03 17:35:11.380535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.380106    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"24096d18dcd946f389f4d67a2aa53a5ca00635d9aff6ab6041ceefd5276f16e6"}
Nov 03 17:35:11.380535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.380154    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"046212d981e6425b6de2007da8d84b39af995ac2bc05e191eb3d4c3560f5b5a8"}
Nov 03 17:35:11.380535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.380195    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"175d23b9fbc2c849db5a3812b1b2a0ab37fd64c7f8e6794c6bb3184333804282"}
Nov 03 17:35:11.380535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.380247    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"dabc27b3376f75f7dac24aff33a40b32a32e4d03559e1e6f40de43c56664972a"}
Nov 03 17:35:11.380535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.380284    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b802a022-e188-4411-806c-d8ccb0e22450","Type":"ContainerStarted","Data":"101b02c2f204986922c078ce404cc9a73bfcbc6d5ba6d980a7b8b0a8a4b6f8e2"}
Nov 03 17:35:11.514380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.514327    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.501620385 podStartE2EDuration="3.51431276s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.692375ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:11.513515328 +0000 UTC m=+47.524253412" watchObservedRunningTime="2026-11-03 17:35:11.51431276 +0000 UTC m=+47.525050826"
Nov 03 17:35:11.830693 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.830654    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"]
Nov 03 17:35:11.833221 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.833196    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.887904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.887862    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 17:35:11.901766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.901714    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/99ef5f46-4180-4268-8b66-ae60cb6b6a12-kube-api-access-qrpwv\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.901895 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.901792    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-secret-telemeter-client\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.901895 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.901869    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-metrics-client-ca\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.901964 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.901910    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.901993 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.901987    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-federate-client-tls\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.902022 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.902011    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.902056 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.902046    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-telemeter-client-tls\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.902086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.902064    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-serving-certs-ca-bundle\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:11.938256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.938224    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 17:35:11.938419 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.938284    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 17:35:11.938648 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.938625    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 17:35:11.938648 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.938638    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sqwbn\""
Nov 03 17:35:11.988287 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.988242    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 17:35:11.988287 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.988242    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 17:35:11.993753 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:11.990881    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"]
Nov 03 17:35:12.002510 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002481    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-federate-client-tls\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002520    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002550    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-telemeter-client-tls\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002580    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-serving-certs-ca-bundle\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002635    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/99ef5f46-4180-4268-8b66-ae60cb6b6a12-kube-api-access-qrpwv\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002670    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-secret-telemeter-client\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002980 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002710    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-metrics-client-ca\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.002980 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.002774    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.003655 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.003623    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-serving-certs-ca-bundle\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.003891 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.003864    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-metrics-client-ca\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.004015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.003992    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ef5f46-4180-4268-8b66-ae60cb6b6a12-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.005384 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.005352    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-telemeter-client-tls\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.005473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.005358    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-federate-client-tls\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.005473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.005456    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.005582 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.005560    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/99ef5f46-4180-4268-8b66-ae60cb6b6a12-secret-telemeter-client\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.066352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.066325    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/99ef5f46-4180-4268-8b66-ae60cb6b6a12-kube-api-access-qrpwv\") pod \"telemeter-client-6dd9bb94f-kcnnn\" (UID: \"99ef5f46-4180-4268-8b66-ae60cb6b6a12\") " pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.146344 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.146318    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"
Nov 03 17:35:12.576841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.576755    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn"]
Nov 03 17:35:12.579790 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:35:12.579761    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ef5f46_4180_4268_8b66_ae60cb6b6a12.slice/crio-7ab40a0674c8c35f3d3bad9275f357a0e194ea2cd1284b37674d426b8eefe33e WatchSource:0}: Error finding container 7ab40a0674c8c35f3d3bad9275f357a0e194ea2cd1284b37674d426b8eefe33e: Status 404 returned error can't find the container with id 7ab40a0674c8c35f3d3bad9275f357a0e194ea2cd1284b37674d426b8eefe33e
Nov 03 17:35:12.581924 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:12.581907    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:35:13.391516 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:13.391483    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn" event={"ID":"99ef5f46-4180-4268-8b66-ae60cb6b6a12","Type":"ContainerStarted","Data":"7ab40a0674c8c35f3d3bad9275f357a0e194ea2cd1284b37674d426b8eefe33e"}
Nov 03 17:35:15.401661 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:15.401623    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn" event={"ID":"99ef5f46-4180-4268-8b66-ae60cb6b6a12","Type":"ContainerStarted","Data":"602e02a7bbf416d4ea2cc661dd3b664507a8ad07ceff2626f8947b336a3ff3f4"}
Nov 03 17:35:15.402093 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:15.401713    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn" event={"ID":"99ef5f46-4180-4268-8b66-ae60cb6b6a12","Type":"ContainerStarted","Data":"264d26fdeafeb522582cc771e39370b84bd396f30115eb8a170f053cdabe3438"}
Nov 03 17:35:15.402093 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:15.401747    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn" event={"ID":"99ef5f46-4180-4268-8b66-ae60cb6b6a12","Type":"ContainerStarted","Data":"183290da2160c0e7c50beff4677d48853ae4dc67f2deb7a2feb6e6175b14492c"}
Nov 03 17:35:15.629162 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:15.629106    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dd9bb94f-kcnnn" podStartSLOduration=2.729476103 podStartE2EDuration="4.629092578s" totalImagesPullingTime="1.899616475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:15.628696605 +0000 UTC m=+51.639434734" watchObservedRunningTime="2026-11-03 17:35:15.629092578 +0000 UTC m=+51.639830644"
Nov 03 17:35:16.035862 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.035829    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f8855cb5-s9bl6"]
Nov 03 17:35:16.040996 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.038355    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.118462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.118421    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f8855cb5-s9bl6"]
Nov 03 17:35:16.140492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.140452    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.140818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.140793    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.140953 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.140826    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.140953 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.140865    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.141047 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.140973    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.141047 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.141042    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.141122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.141061    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5w4\" (UniqueName: \"kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.141122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.141111    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.241615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.241575    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.241826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.241624    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.241826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.241668    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.241826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.241694    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5w4\" (UniqueName: \"kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.241826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.241719    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242027    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242147 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242123    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242159    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242174    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242443    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242534    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242621    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.242800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.242780    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.244807 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.244789    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.244867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.244849    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.305871 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.305800    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5w4\" (UniqueName: \"kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4\") pod \"console-76f8855cb5-s9bl6\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") " pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.328392 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.328363    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:16.349527 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.349475    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:16.497079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.497038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:16.726819 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.726761    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:16.777615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.777585    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f8855cb5-s9bl6"]
Nov 03 17:35:16.780430 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:35:16.780397    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb9f401_08c2_45a7_83cf_b32f9c171f54.slice/crio-4d5a25eecab42982ccf57b3f96f08f32890451b3eb38f2fcf46d3bcec825c027 WatchSource:0}: Error finding container 4d5a25eecab42982ccf57b3f96f08f32890451b3eb38f2fcf46d3bcec825c027: Status 404 returned error can't find the container with id 4d5a25eecab42982ccf57b3f96f08f32890451b3eb38f2fcf46d3bcec825c027
Nov 03 17:35:16.883880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:16.883844    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:17.412076 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:17.412042    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f8855cb5-s9bl6" event={"ID":"1eb9f401-08c2-45a7-83cf-b32f9c171f54","Type":"ContainerStarted","Data":"e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c"}
Nov 03 17:35:17.412262 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:17.412107    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f8855cb5-s9bl6" event={"ID":"1eb9f401-08c2-45a7-83cf-b32f9c171f54","Type":"ContainerStarted","Data":"4d5a25eecab42982ccf57b3f96f08f32890451b3eb38f2fcf46d3bcec825c027"}
Nov 03 17:35:17.536565 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:17.536517    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76f8855cb5-s9bl6" podStartSLOduration=2.5365021199999997 podStartE2EDuration="2.53650212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:17.536140197 +0000 UTC m=+53.546878272" watchObservedRunningTime="2026-11-03 17:35:17.53650212 +0000 UTC m=+53.547240189"
Nov 03 17:35:26.350609 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:26.350563    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:26.351102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:26.350625    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:26.355507 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:26.355482    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:26.450324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:26.450295    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:35:26.639015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:26.638984    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d7f486f8-7vxp2"]
Nov 03 17:35:30.636352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:30.636319    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:36.208620 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:36.208546    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:40.338146 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:40.338111    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:35:51.662861 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.662819    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86d7f486f8-7vxp2" podUID="5f6bc412-5a3c-4826-a2df-123014d49371" containerName="console" containerID="cri-o://fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3" gracePeriod=15
Nov 03 17:35:51.924324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.924245    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:35:51.944799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.944770    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.944970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.944815    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.944970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.944835    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.944970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.944856    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8\" (UniqueName: \"kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.945128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945041    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.945128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945094    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.945233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945139    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.945233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945172    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:35:51.945233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945185    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert\") pod \"5f6bc412-5a3c-4826-a2df-123014d49371\" (UID: \"5f6bc412-5a3c-4826-a2df-123014d49371\") "
Nov 03 17:35:51.945378 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945226    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:51.945435 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945404    5838 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-console-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:51.945435 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945423    5838 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f6bc412-5a3c-4826-a2df-123014d49371-tmp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:51.945661 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945637    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:51.945723 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945709    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:51.945798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.945706    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:51.947120 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.947101    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:51.947671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.947648    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:51.947671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:51.947657    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8" pod "5f6bc412-5a3c-4826-a2df-123014d49371" (UID: "5f6bc412-5a3c-4826-a2df-123014d49371"). InnerVolumeSpecName "kube-api-access-phlq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:52.046290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.046175    5838 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-oauth-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:52.046290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.046208    5838 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:52.046290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.046222    5838 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-trusted-ca-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:52.046290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.046236    5838 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f6bc412-5a3c-4826-a2df-123014d49371-oauth-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:52.046290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.046250    5838 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f6bc412-5a3c-4826-a2df-123014d49371-console-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:52.046290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.046264    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phlq8\" (UniqueName: \"kubernetes.io/projected/5f6bc412-5a3c-4826-a2df-123014d49371-kube-api-access-phlq8\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:52.547831 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.547799    5838 generic.go:356] "Generic (PLEG): container finished" podID="5f6bc412-5a3c-4826-a2df-123014d49371" containerID="fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3" exitCode=0
Nov 03 17:35:52.548011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.547849    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d7f486f8-7vxp2"
Nov 03 17:35:52.548011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.547858    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d7f486f8-7vxp2" event={"ID":"5f6bc412-5a3c-4826-a2df-123014d49371","Type":"ContainerDied","Data":"fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3"}
Nov 03 17:35:52.548011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.547900    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d7f486f8-7vxp2" event={"ID":"5f6bc412-5a3c-4826-a2df-123014d49371","Type":"ContainerDied","Data":"1d4415cfaa4b885d8e73e1ba8145e97ac787328baae7bbdcb215bcc5f99c6a92"}
Nov 03 17:35:52.548011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.547923    5838 scope.go:122] "RemoveContainer" containerID="fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3"
Nov 03 17:35:52.556077 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.556050    5838 scope.go:122] "RemoveContainer" containerID="fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3"
Nov 03 17:35:52.556385 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:35:52.556364    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3\": container with ID starting with fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3 not found: ID does not exist" containerID="fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3"
Nov 03 17:35:52.556470 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.556400    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3"} err="failed to get container status \"fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3\": rpc error: code = NotFound desc = could not find container \"fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3\": container with ID starting with fbc590d03771d1ae710f05000ac17a99e562926b451ff530b73b5c0804d69be3 not found: ID does not exist"
Nov 03 17:35:52.715380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.715342    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d7f486f8-7vxp2"]
Nov 03 17:35:52.766054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:52.766024    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86d7f486f8-7vxp2"]
Nov 03 17:35:54.113008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:35:54.112977    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6bc412-5a3c-4826-a2df-123014d49371" path="/var/lib/kubelet/pods/5f6bc412-5a3c-4826-a2df-123014d49371/volumes"
Nov 03 17:36:10.338185 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:36:10.338106    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:36:19.098231 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:19.098167    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:19.265859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:19.265820    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:23.765638 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:23.765561    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:23.940844 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:23.940814    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:25.474640 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.474607    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5455cbb8c6-bzvmh"]
Nov 03 17:36:25.475058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.474981    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5f6bc412-5a3c-4826-a2df-123014d49371" containerName="console"
Nov 03 17:36:25.475058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.474993    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6bc412-5a3c-4826-a2df-123014d49371" containerName="console"
Nov 03 17:36:25.475058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.475049    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6bc412-5a3c-4826-a2df-123014d49371" containerName="console"
Nov 03 17:36:25.479211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.479195    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.540035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.540005    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5455cbb8c6-bzvmh"]
Nov 03 17:36:25.628455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628414    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4445b143-5b85-4864-87a0-099d4a9c451b-tmp\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628456    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-console-config\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628477    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-service-ca\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628546    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss8qv\" (UniqueName: \"kubernetes.io/projected/4445b143-5b85-4864-87a0-099d4a9c451b-kube-api-access-ss8qv\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628670    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-oauth-serving-cert\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628704    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-trusted-ca-bundle\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628817    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4445b143-5b85-4864-87a0-099d4a9c451b-console-serving-cert\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.628939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.628849    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4445b143-5b85-4864-87a0-099d4a9c451b-console-oauth-config\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729375    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-oauth-serving-cert\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729419    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-trusted-ca-bundle\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729449    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4445b143-5b85-4864-87a0-099d4a9c451b-console-serving-cert\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729467    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4445b143-5b85-4864-87a0-099d4a9c451b-console-oauth-config\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729520    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4445b143-5b85-4864-87a0-099d4a9c451b-tmp\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729543    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-console-config\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729573    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-service-ca\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.729906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.729595    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss8qv\" (UniqueName: \"kubernetes.io/projected/4445b143-5b85-4864-87a0-099d4a9c451b-kube-api-access-ss8qv\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.730431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.730221    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4445b143-5b85-4864-87a0-099d4a9c451b-tmp\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.730431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.730273    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-oauth-serving-cert\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.730607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.730491    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-console-config\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.730667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.730641    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-service-ca\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.730981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.730958    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4445b143-5b85-4864-87a0-099d4a9c451b-trusted-ca-bundle\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.732229 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.732206    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4445b143-5b85-4864-87a0-099d4a9c451b-console-oauth-config\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.732341 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.732324    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4445b143-5b85-4864-87a0-099d4a9c451b-console-serving-cert\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:25.761158 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.761120    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:25.793278 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:25.793250    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss8qv\" (UniqueName: \"kubernetes.io/projected/4445b143-5b85-4864-87a0-099d4a9c451b-kube-api-access-ss8qv\") pod \"console-5455cbb8c6-bzvmh\" (UID: \"4445b143-5b85-4864-87a0-099d4a9c451b\") " pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:26.090804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.090688    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:26.173674 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.173623    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:26.362271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.362200    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:26.468303 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.468264    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5455cbb8c6-bzvmh"]
Nov 03 17:36:26.473065 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:36:26.473023    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4445b143_5b85_4864_87a0_099d4a9c451b.slice/crio-7ad5ca7f433dee76d0ebdbf7afcf9b4ec1bbf462b2989b6bfd17c3d9c2530867 WatchSource:0}: Error finding container 7ad5ca7f433dee76d0ebdbf7afcf9b4ec1bbf462b2989b6bfd17c3d9c2530867: Status 404 returned error can't find the container with id 7ad5ca7f433dee76d0ebdbf7afcf9b4ec1bbf462b2989b6bfd17c3d9c2530867
Nov 03 17:36:26.675900 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.675870    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5455cbb8c6-bzvmh" event={"ID":"4445b143-5b85-4864-87a0-099d4a9c451b","Type":"ContainerStarted","Data":"5df1486664f8d8f7b0252b289bdd3530a4379f1b2728eef81101672fcde87624"}
Nov 03 17:36:26.676318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.675917    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5455cbb8c6-bzvmh" event={"ID":"4445b143-5b85-4864-87a0-099d4a9c451b","Type":"ContainerStarted","Data":"7ad5ca7f433dee76d0ebdbf7afcf9b4ec1bbf462b2989b6bfd17c3d9c2530867"}
Nov 03 17:36:26.798004 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:26.797950    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5455cbb8c6-bzvmh" podStartSLOduration=1.797935212 podStartE2EDuration="1.797935212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:26.797374416 +0000 UTC m=+122.808112488" watchObservedRunningTime="2026-11-03 17:36:26.797935212 +0000 UTC m=+122.808673278"
Nov 03 17:36:30.978910 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:30.978872    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:36.091429 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.091389    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:36.091886 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.091452    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:36.096302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.096277    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:36.409081 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.409044    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:36.567039 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.567001    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:36.716438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.716354    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5455cbb8c6-bzvmh"
Nov 03 17:36:36.926752 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:36.926696    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f8855cb5-s9bl6"]
Nov 03 17:36:40.337056 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:36:40.337021    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:36:41.652724 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:41.652686    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:46.932987 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:46.932940    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:47.139105 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:36:47.139069    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:01.948000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:01.947952    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76f8855cb5-s9bl6" podUID="1eb9f401-08c2-45a7-83cf-b32f9c171f54" containerName="console" containerID="cri-o://e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c" gracePeriod=15
Nov 03 17:37:02.198296 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.198221    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:37:02.241492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241462    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4\" (UniqueName: \"kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.241672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241511    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.241672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241567    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.241672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241585    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.241672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241611    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.241672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241639    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.242232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241711    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.242232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.241773    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp\") pod \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\" (UID: \"1eb9f401-08c2-45a7-83cf-b32f9c171f54\") "
Nov 03 17:37:02.242232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.242130    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:02.242232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.242156    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:02.242612 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.242139    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:02.242612 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.242259    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:37:02.242951 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.242920    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:02.244005 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.243982    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "kube-api-access-zr5w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:02.244096 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.244011    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:02.244096 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.244029    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert" pod "1eb9f401-08c2-45a7-83cf-b32f9c171f54" (UID: "1eb9f401-08c2-45a7-83cf-b32f9c171f54"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:02.343075 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343044    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr5w4\" (UniqueName: \"kubernetes.io/projected/1eb9f401-08c2-45a7-83cf-b32f9c171f54-kube-api-access-zr5w4\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343075 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343073    5838 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-oauth-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343085    5838 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343094    5838 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-trusted-ca-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343102    5838 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343111    5838 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-oauth-serving-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343119    5838 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb9f401-08c2-45a7-83cf-b32f9c171f54-console-config\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.343280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.343128    5838 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb9f401-08c2-45a7-83cf-b32f9c171f54-tmp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.807188 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.807155    5838 generic.go:356] "Generic (PLEG): container finished" podID="1eb9f401-08c2-45a7-83cf-b32f9c171f54" containerID="e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c" exitCode=0
Nov 03 17:37:02.807373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.807203    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f8855cb5-s9bl6"
Nov 03 17:37:02.807373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.807221    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f8855cb5-s9bl6" event={"ID":"1eb9f401-08c2-45a7-83cf-b32f9c171f54","Type":"ContainerDied","Data":"e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c"}
Nov 03 17:37:02.807373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.807263    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f8855cb5-s9bl6" event={"ID":"1eb9f401-08c2-45a7-83cf-b32f9c171f54","Type":"ContainerDied","Data":"4d5a25eecab42982ccf57b3f96f08f32890451b3eb38f2fcf46d3bcec825c027"}
Nov 03 17:37:02.807373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.807286    5838 scope.go:122] "RemoveContainer" containerID="e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c"
Nov 03 17:37:02.815913 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.815897    5838 scope.go:122] "RemoveContainer" containerID="e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c"
Nov 03 17:37:02.816234 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:37:02.816214    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c\": container with ID starting with e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c not found: ID does not exist" containerID="e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c"
Nov 03 17:37:02.816299 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.816248    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c"} err="failed to get container status \"e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c\": rpc error: code = NotFound desc = could not find container \"e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c\": container with ID starting with e80a9f176b3105827c6d0ecda3f253f8445d318584b87e70e74257c1d506b09c not found: ID does not exist"
Nov 03 17:37:02.977067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:02.977033    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f8855cb5-s9bl6"]
Nov 03 17:37:03.027295 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:03.027257    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76f8855cb5-s9bl6"]
Nov 03 17:37:04.114080 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:04.114044    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb9f401-08c2-45a7-83cf-b32f9c171f54" path="/var/lib/kubelet/pods/1eb9f401-08c2-45a7-83cf-b32f9c171f54/volumes"
Nov 03 17:37:10.337624 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:37:10.337581    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:37:31.344202 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:37:31.344160    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:40.336917 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:37:40.336874    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:38:10.337969 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:38:10.337930    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:38:31.681323 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:38:31.681285    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:40.338510 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:38:40.338460    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:39:02.329198 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:02.329163    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:10.337190 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:39:10.337149    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:39:24.173071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:24.173044    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:39:24.175262 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:24.175244    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:39:31.502137 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:31.502101    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:31.688918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:31.688878    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:32.027660 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.027620    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:32.514102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.514062    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"]
Nov 03 17:39:32.520138 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.520098    5838 cpu_manager.go:414] "containerMap: removing container" podUID="1eb9f401-08c2-45a7-83cf-b32f9c171f54" containerName="console"
Nov 03 17:39:32.520138 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.520134    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb9f401-08c2-45a7-83cf-b32f9c171f54" containerName="console"
Nov 03 17:39:32.520430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.520418    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb9f401-08c2-45a7-83cf-b32f9c171f54" containerName="console"
Nov 03 17:39:32.523640 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.523621    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.613774 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.613721    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 17:39:32.614022 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.613813    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 17:39:32.614101 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.614032    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-j275m\""
Nov 03 17:39:32.614904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.614871    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"]
Nov 03 17:39:32.683453 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.683420    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.683668 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.683484    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qxr\" (UniqueName: \"kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.683757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.683703    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.700226 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.700197    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"]
Nov 03 17:39:32.703647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.703629    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.761946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.761885    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-dqs8l\""
Nov 03 17:39:32.769352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.769277    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"]
Nov 03 17:39:32.784192 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.784169    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.784353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.784211    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.784353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.784241    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxr\" (UniqueName: \"kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.784633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.784611    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.784680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.784620    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.847542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.847512    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qxr\" (UniqueName: \"kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:32.885623 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.885577    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gv9\" (UniqueName: \"kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.885821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.885646    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.885821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.885770    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.986602 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.986565    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.986762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.986626    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gv9\" (UniqueName: \"kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.986762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.986673    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.987053 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.987026    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:32.987134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:32.987082    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:33.050574 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.050506    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gv9\" (UniqueName: \"kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:33.139622 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.139584    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:39:33.318725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.318638    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:39:33.351845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.351815    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:33.516574 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.516541    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"]
Nov 03 17:39:33.517095 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.516990    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:33.520098 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:33.520075    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aeab935_b169_4f94_988f_58928f6fd468.slice/crio-3282b98ca5208d9d38f52b85ae3058876d64f939b768fb9219132fa41507d138 WatchSource:0}: Error finding container 3282b98ca5208d9d38f52b85ae3058876d64f939b768fb9219132fa41507d138: Status 404 returned error can't find the container with id 3282b98ca5208d9d38f52b85ae3058876d64f939b768fb9219132fa41507d138
Nov 03 17:39:33.696834 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:33.696801    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"]
Nov 03 17:39:33.700141 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:33.700116    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ba4520_12ef_4c93_8d7c_0ee9960a3ada.slice/crio-a94ac969d6a3a11941c7af298d2c157f810dceaa8a946140d20097adcb35f955 WatchSource:0}: Error finding container a94ac969d6a3a11941c7af298d2c157f810dceaa8a946140d20097adcb35f955: Status 404 returned error can't find the container with id a94ac969d6a3a11941c7af298d2c157f810dceaa8a946140d20097adcb35f955
Nov 03 17:39:34.359009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:34.358970    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerStarted","Data":"a94ac969d6a3a11941c7af298d2c157f810dceaa8a946140d20097adcb35f955"}
Nov 03 17:39:34.360131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:34.360108    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r" event={"ID":"5aeab935-b169-4f94-988f-58928f6fd468","Type":"ContainerStarted","Data":"3282b98ca5208d9d38f52b85ae3058876d64f939b768fb9219132fa41507d138"}
Nov 03 17:39:34.967279 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:34.967156    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:36.726630 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.726593    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"]
Nov 03 17:39:36.730818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.730788    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.820423 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.820370    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.820611 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.820441    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgq5\" (UniqueName: \"kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.820611 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.820587    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.835229 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.835174    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-j8jdj\""
Nov 03 17:39:36.836314 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.836284    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"]
Nov 03 17:39:36.922090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.922054    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgq5\" (UniqueName: \"kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.922350 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.922147    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.922350 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.922208    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.922647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.922625    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.922696 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.922648    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:36.986605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:36.986519    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgq5\" (UniqueName: \"kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:37.051900 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:37.051859    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:39:37.441703 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:37.441666    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"]
Nov 03 17:39:37.479954 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:37.479914    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ca8273_9d83_4643_8e42_860d846dfd7d.slice/crio-a543a20dd057f7e554a3f505717717046b9911d1ce48671b6b943ad59801b47b WatchSource:0}: Error finding container a543a20dd057f7e554a3f505717717046b9911d1ce48671b6b943ad59801b47b: Status 404 returned error can't find the container with id a543a20dd057f7e554a3f505717717046b9911d1ce48671b6b943ad59801b47b
Nov 03 17:39:38.007499 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.007464    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"]
Nov 03 17:39:38.011682 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.011657    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.065504 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.065454    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-whp67\""
Nov 03 17:39:38.102052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.102017    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"]
Nov 03 17:39:38.105915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.105892    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.108306 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.108276    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"]
Nov 03 17:39:38.132874 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.132841    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55rt\" (UniqueName: \"kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.133053 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.132973    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.133053 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.133020    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.160004 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.159950    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qc8kp\""
Nov 03 17:39:38.165667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.165631    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"]
Nov 03 17:39:38.204632 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.204597    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"]
Nov 03 17:39:38.208850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.208821    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.234392 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234356    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.234552 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234436    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.234552 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234477    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.234691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234596    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.234777 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234684    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsb2\" (UniqueName: \"kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.234777 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234758    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55rt\" (UniqueName: \"kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.234914 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234893    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.234960 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.234931    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.264022 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.263941    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rvrjc\""
Nov 03 17:39:38.269133 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.269095    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"]
Nov 03 17:39:38.300619 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.300588    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"]
Nov 03 17:39:38.301049 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.300959    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55rt\" (UniqueName: \"kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.305162 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.305129    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.330418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.330389    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:39:38.335575 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.335544    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsb2\" (UniqueName: \"kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.335760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.335612    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.335760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.335638    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.335760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.335664    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.335961 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.335806    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.335961 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.335888    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp89\" (UniqueName: \"kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.336242 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.336219    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.336317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.336227    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.365176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.365124    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-s4b4w\""
Nov 03 17:39:38.365428 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.365354    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"]
Nov 03 17:39:38.379218 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.379153    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" event={"ID":"55ca8273-9d83-4643-8e42-860d846dfd7d","Type":"ContainerStarted","Data":"a543a20dd057f7e554a3f505717717046b9911d1ce48671b6b943ad59801b47b"}
Nov 03 17:39:38.407038 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.407010    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsb2\" (UniqueName: \"kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.426352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.425171    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:39:38.437355 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.437325    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.437563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.437359    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.437563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.437410    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmnx\" (UniqueName: \"kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.437563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.437439    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp89\" (UniqueName: \"kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.437563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.437465    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.437563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.437482    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.438150 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.438086    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.438379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.438307    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.505871 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.505303    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp89\" (UniqueName: \"kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.527663 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.527366    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:39:38.538776 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.537953    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.538776 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.537997    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.538776 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.538109    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmnx\" (UniqueName: \"kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.538776 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.538446    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.538776 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.538693    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.604257 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.604201    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmnx\" (UniqueName: \"kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.630012 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.629187    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:39:38.721370 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.721329    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"]
Nov 03 17:39:38.725362 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:38.725326    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b009f5_0030_4c83_853c_cd8f56060540.slice/crio-88cff690fdbe8383865fa25a130173b2d142c1e725f7b3f49d30c7838cbfd17a WatchSource:0}: Error finding container 88cff690fdbe8383865fa25a130173b2d142c1e725f7b3f49d30c7838cbfd17a: Status 404 returned error can't find the container with id 88cff690fdbe8383865fa25a130173b2d142c1e725f7b3f49d30c7838cbfd17a
Nov 03 17:39:38.832927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.832865    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"]
Nov 03 17:39:38.836847 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:38.836805    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe82289f_0ef4_4ea9_b918_c2db1034ae34.slice/crio-c6d33d972b01805ecb74666c94f0dd91b5be10993bb9aabb6938de654816e1eb WatchSource:0}: Error finding container c6d33d972b01805ecb74666c94f0dd91b5be10993bb9aabb6938de654816e1eb: Status 404 returned error can't find the container with id c6d33d972b01805ecb74666c94f0dd91b5be10993bb9aabb6938de654816e1eb
Nov 03 17:39:38.950910 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:38.950874    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"]
Nov 03 17:39:39.031149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.031036    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"]
Nov 03 17:39:39.034695 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:39.034644    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93729b8a_ded8_4ddc_ae53_da8267fd201e.slice/crio-4c6130d474320ce3530f806a1de3af9d96b165080af12bc84c3119f95e91348f WatchSource:0}: Error finding container 4c6130d474320ce3530f806a1de3af9d96b165080af12bc84c3119f95e91348f: Status 404 returned error can't find the container with id 4c6130d474320ce3530f806a1de3af9d96b165080af12bc84c3119f95e91348f
Nov 03 17:39:39.387438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.387343    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk" event={"ID":"56fc86bb-10c1-428c-9829-256bfea6f746","Type":"ContainerStarted","Data":"2fe42f3e27620935e68b90bfc191529ae5b689e9150dc76636f85e4b5247fb81"}
Nov 03 17:39:39.388924 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.388891    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng" event={"ID":"fe82289f-0ef4-4ea9-b918-c2db1034ae34","Type":"ContainerStarted","Data":"c6d33d972b01805ecb74666c94f0dd91b5be10993bb9aabb6938de654816e1eb"}
Nov 03 17:39:39.391016 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.390956    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9" event={"ID":"93729b8a-ded8-4ddc-ae53-da8267fd201e","Type":"ContainerStarted","Data":"4c6130d474320ce3530f806a1de3af9d96b165080af12bc84c3119f95e91348f"}
Nov 03 17:39:39.393335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.393305    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp" event={"ID":"65b009f5-0030-4c83-853c-cd8f56060540","Type":"ContainerStarted","Data":"88cff690fdbe8383865fa25a130173b2d142c1e725f7b3f49d30c7838cbfd17a"}
Nov 03 17:39:39.682300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.682263    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"]
Nov 03 17:39:39.686477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.686452    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.747129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.747091    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"]
Nov 03 17:39:39.758393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.758140    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkw2\" (UniqueName: \"kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.758393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.758196    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.758393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.758277    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.858962 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.858919    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkw2\" (UniqueName: \"kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.859134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.858991    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.859134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.859110    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.859607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.859583    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.860080 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.860055    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:39.928686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:39.928647    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkw2\" (UniqueName: \"kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:40.008970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.008610    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:39:40.077550 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.077481    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"]
Nov 03 17:39:40.082701 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.082669    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.142522 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.142491    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"]
Nov 03 17:39:40.262657 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.262568    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.262657 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.262629    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5b9p\" (UniqueName: \"kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.262921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.262717    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.355123 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:39:40.355051    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:39:40.365448 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.364167    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.365448 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.364216    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b9p\" (UniqueName: \"kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.365448 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.364278    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.365448 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.364780    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.365448 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.365035    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.436511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.436419    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b9p\" (UniqueName: \"kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:40.449235 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.449116    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"]
Nov 03 17:39:40.452872 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:40.452831    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef56dac_91fe_41d4_b5be_db45b5844268.slice/crio-b36de7c8259e80899d6c8fa860fcf12b7b47e1b93ad9bb68ac59648bf410e69a WatchSource:0}: Error finding container b36de7c8259e80899d6c8fa860fcf12b7b47e1b93ad9bb68ac59648bf410e69a: Status 404 returned error can't find the container with id b36de7c8259e80899d6c8fa860fcf12b7b47e1b93ad9bb68ac59648bf410e69a
Nov 03 17:39:40.709095 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:40.708819    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:39:41.134601 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:41.134208    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"]
Nov 03 17:39:41.160553 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:41.160511    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc84aa91_c637_4481_a958_9598a0f4913b.slice/crio-46a868b2da3e2cec6226e4d52881228b40a96cc979bdf44f45351124f4e274a2 WatchSource:0}: Error finding container 46a868b2da3e2cec6226e4d52881228b40a96cc979bdf44f45351124f4e274a2: Status 404 returned error can't find the container with id 46a868b2da3e2cec6226e4d52881228b40a96cc979bdf44f45351124f4e274a2
Nov 03 17:39:41.411703 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:41.411608    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" event={"ID":"bc84aa91-c637-4481-a958-9598a0f4913b","Type":"ContainerStarted","Data":"46a868b2da3e2cec6226e4d52881228b40a96cc979bdf44f45351124f4e274a2"}
Nov 03 17:39:41.415282 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:41.415198    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" event={"ID":"6ef56dac-91fe-41d4-b5be-db45b5844268","Type":"ContainerStarted","Data":"b36de7c8259e80899d6c8fa860fcf12b7b47e1b93ad9bb68ac59648bf410e69a"}
Nov 03 17:39:42.428483 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:42.428416    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerStarted","Data":"8b0693adbe8c05952cd8ed0e4bb9af348c16ee60011ea24070b052347d0152f8"}
Nov 03 17:39:43.443933 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:43.443891    5838 generic.go:356] "Generic (PLEG): container finished" podID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerID="8b0693adbe8c05952cd8ed0e4bb9af348c16ee60011ea24070b052347d0152f8" exitCode=0
Nov 03 17:39:43.444450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:43.443993    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerDied","Data":"8b0693adbe8c05952cd8ed0e4bb9af348c16ee60011ea24070b052347d0152f8"}
Nov 03 17:39:43.901164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:43.901048    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:44.080557 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:44.080517    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:44.652483 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:44.652436    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:44.830495 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:44.830455    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:45.304102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.304062    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"]
Nov 03 17:39:45.308264 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.308235    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.363667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.363634    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\""
Nov 03 17:39:45.384672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.384639    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 17:39:45.384890 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.384857    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 17:39:45.412647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.412583    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 17:39:45.414352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.414326    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"]
Nov 03 17:39:45.414866 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.414844    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:45.422887 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.422850    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423024 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.422916    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423024 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.422946    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423024 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.423000    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423214 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.423039    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423214 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.423075    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423214 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.423107    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423214 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.423145    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.423214 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.423177    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkgk\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.468621 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.467538    5838 generic.go:356] "Generic (PLEG): container finished" podID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerID="c57efe5e1085bea31be43c191ff0836710727469f9c4700b6c430f58b5712750" exitCode=0
Nov 03 17:39:45.468621 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.467691    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerDied","Data":"c57efe5e1085bea31be43c191ff0836710727469f9c4700b6c430f58b5712750"}
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524024    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkgk\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524107    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524156    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524186    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524249    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524285    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524320    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524350    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.524838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.524381    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.525673 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.525510    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.526604 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.526580    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.526762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.526724    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.526880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.526863    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.527691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.527647    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.551040 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.550968    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.554853 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.554780    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.603122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.603087    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.607139 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.607075    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkgk\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jv82l\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.631431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.630662    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:45.632105 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.632016    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:45.643130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.643099    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"]
Nov 03 17:39:45.656198 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.654983    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.711909 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.711805    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-5q8s4\""
Nov 03 17:39:45.713948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.713818    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"]
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726437    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726494    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cad14b8-057f-416d-938c-505f90358b9a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726547    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726576    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726607    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726658    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cad14b8-057f-416d-938c-505f90358b9a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726723    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cad14b8-057f-416d-938c-505f90358b9a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726786    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.727074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.726818    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9k2p\" (UniqueName: \"kubernetes.io/projected/7cad14b8-057f-416d-938c-505f90358b9a-kube-api-access-b9k2p\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.792296 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.792193    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827565    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9k2p\" (UniqueName: \"kubernetes.io/projected/7cad14b8-057f-416d-938c-505f90358b9a-kube-api-access-b9k2p\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827635    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827678    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cad14b8-057f-416d-938c-505f90358b9a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827724    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827773    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827802    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827855    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cad14b8-057f-416d-938c-505f90358b9a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827920    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cad14b8-057f-416d-938c-505f90358b9a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.827970    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.828395    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.828994    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.829578    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cad14b8-057f-416d-938c-505f90358b9a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.830567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.829833    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.831913 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.831613    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.896168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.896095    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cad14b8-057f-416d-938c-505f90358b9a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.899619 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.899523    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9k2p\" (UniqueName: \"kubernetes.io/projected/7cad14b8-057f-416d-938c-505f90358b9a-kube-api-access-b9k2p\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.900405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.900281    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cad14b8-057f-416d-938c-505f90358b9a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.904129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.904078    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cad14b8-057f-416d-938c-505f90358b9a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx\" (UID: \"7cad14b8-057f-416d-938c-505f90358b9a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:45.995412 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:45.995089    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:46.156817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:46.156696    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"]
Nov 03 17:39:46.461473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:46.460280    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"]
Nov 03 17:39:46.467402 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:39:46.467345    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cad14b8_057f_416d_938c_505f90358b9a.slice/crio-faeb665aaa4ea59331a7cbe3935689c1f262e2dbfff4e64f090c8e5c9d2eeaa8 WatchSource:0}: Error finding container faeb665aaa4ea59331a7cbe3935689c1f262e2dbfff4e64f090c8e5c9d2eeaa8: Status 404 returned error can't find the container with id faeb665aaa4ea59331a7cbe3935689c1f262e2dbfff4e64f090c8e5c9d2eeaa8
Nov 03 17:39:46.510343 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:46.510302    5838 generic.go:356] "Generic (PLEG): container finished" podID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerID="40c483a8f6fb61215cc5a07cd381949d80094587f9765a5cb8b4e169ddfd892c" exitCode=0
Nov 03 17:39:46.510500 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:46.510467    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" event={"ID":"55ca8273-9d83-4643-8e42-860d846dfd7d","Type":"ContainerDied","Data":"40c483a8f6fb61215cc5a07cd381949d80094587f9765a5cb8b4e169ddfd892c"}
Nov 03 17:39:46.518876 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:46.518831    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx" event={"ID":"7cad14b8-057f-416d-938c-505f90358b9a","Type":"ContainerStarted","Data":"faeb665aaa4ea59331a7cbe3935689c1f262e2dbfff4e64f090c8e5c9d2eeaa8"}
Nov 03 17:39:46.521073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:46.521046    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" event={"ID":"3f4f6054-05f5-41c5-848f-b245cbff25d3","Type":"ContainerStarted","Data":"5c566f83c4974016427d393ebd6d59909b68d726f7ee4bd0eaddb443bf839859"}
Nov 03 17:39:47.337368 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:47.337315    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:47.497241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:47.497205    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:48.549724 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:48.549670    5838 generic.go:356] "Generic (PLEG): container finished" podID="5aeab935-b169-4f94-988f-58928f6fd468" containerID="0c80940336cf00dd2e96721b698ff46bc7a851c452294d761de2404a36e605b3" exitCode=0
Nov 03 17:39:48.550393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:48.549890    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r" event={"ID":"5aeab935-b169-4f94-988f-58928f6fd468","Type":"ContainerDied","Data":"0c80940336cf00dd2e96721b698ff46bc7a851c452294d761de2404a36e605b3"}
Nov 03 17:39:49.243233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.243149    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:49.569781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.569678    5838 generic.go:356] "Generic (PLEG): container finished" podID="56fc86bb-10c1-428c-9829-256bfea6f746" containerID="318bdb62128fea9c0f13d5dac20dcc89c2f7651889adc7f5b630d45e661ecd57" exitCode=0
Nov 03 17:39:49.570314 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.569922    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk" event={"ID":"56fc86bb-10c1-428c-9829-256bfea6f746","Type":"ContainerDied","Data":"318bdb62128fea9c0f13d5dac20dcc89c2f7651889adc7f5b630d45e661ecd57"}
Nov 03 17:39:49.574417 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.574386    5838 generic.go:356] "Generic (PLEG): container finished" podID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerID="c28ffc72aef21ee1e3bc4f1b9d8addb7f056a5bfa8c47d79f1d47f001228fcdb" exitCode=0
Nov 03 17:39:49.574542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.574481    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9" event={"ID":"93729b8a-ded8-4ddc-ae53-da8267fd201e","Type":"ContainerDied","Data":"c28ffc72aef21ee1e3bc4f1b9d8addb7f056a5bfa8c47d79f1d47f001228fcdb"}
Nov 03 17:39:49.579699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.578877    5838 generic.go:356] "Generic (PLEG): container finished" podID="65b009f5-0030-4c83-853c-cd8f56060540" containerID="64373b9444637b8ea00a7762952c96782975c7d5b876df70f9efd4f3c9ea2a1b" exitCode=0
Nov 03 17:39:49.579699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:49.578995    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp" event={"ID":"65b009f5-0030-4c83-853c-cd8f56060540","Type":"ContainerDied","Data":"64373b9444637b8ea00a7762952c96782975c7d5b876df70f9efd4f3c9ea2a1b"}
Nov 03 17:39:50.592078 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:50.592040    5838 generic.go:356] "Generic (PLEG): container finished" podID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerID="e2b1efaa42ebd3946d7f2eb687bbbd3eefe3609933c33c60244e74782b237716" exitCode=0
Nov 03 17:39:50.592649 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:50.592186    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng" event={"ID":"fe82289f-0ef4-4ea9-b918-c2db1034ae34","Type":"ContainerDied","Data":"e2b1efaa42ebd3946d7f2eb687bbbd3eefe3609933c33c60244e74782b237716"}
Nov 03 17:39:51.617233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:51.616866    5838 generic.go:356] "Generic (PLEG): container finished" podID="56fc86bb-10c1-428c-9829-256bfea6f746" containerID="ce712c74710ad8002de1a352d26b1f5441ec13bfad3e6b919fecee3fd9022c66" exitCode=0
Nov 03 17:39:51.617233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:51.616984    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk" event={"ID":"56fc86bb-10c1-428c-9829-256bfea6f746","Type":"ContainerDied","Data":"ce712c74710ad8002de1a352d26b1f5441ec13bfad3e6b919fecee3fd9022c66"}
Nov 03 17:39:51.631677 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:51.631435    5838 generic.go:356] "Generic (PLEG): container finished" podID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerID="f1d4b7fb10912892326256e56100ac3917624bc3e2441f5bb31d3e82e13682c9" exitCode=0
Nov 03 17:39:51.631677 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:51.631544    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" event={"ID":"6ef56dac-91fe-41d4-b5be-db45b5844268","Type":"ContainerDied","Data":"f1d4b7fb10912892326256e56100ac3917624bc3e2441f5bb31d3e82e13682c9"}
Nov 03 17:39:51.636301 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:51.636260    5838 generic.go:356] "Generic (PLEG): container finished" podID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerID="4a349bb822907bebd25d1494fcd36303096017d280357e86ed4e7f4a4d52191c" exitCode=0
Nov 03 17:39:51.636387 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:51.636324    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9" event={"ID":"93729b8a-ded8-4ddc-ae53-da8267fd201e","Type":"ContainerDied","Data":"4a349bb822907bebd25d1494fcd36303096017d280357e86ed4e7f4a4d52191c"}
Nov 03 17:39:52.652978 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:52.652775    5838 generic.go:356] "Generic (PLEG): container finished" podID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerID="b36f333e4a5592e9efd94890882e631f4fa7faeda194a2f58ab2909a3d35879e" exitCode=0
Nov 03 17:39:52.652978 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:52.652906    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng" event={"ID":"fe82289f-0ef4-4ea9-b918-c2db1034ae34","Type":"ContainerDied","Data":"b36f333e4a5592e9efd94890882e631f4fa7faeda194a2f58ab2909a3d35879e"}
Nov 03 17:39:52.661853 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:52.659098    5838 generic.go:356] "Generic (PLEG): container finished" podID="65b009f5-0030-4c83-853c-cd8f56060540" containerID="40cd027dd16e8fc265dc39c382ecdd64211a1e50ef0dc745696e5b1b8b188964" exitCode=0
Nov 03 17:39:52.661853 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:52.659248    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp" event={"ID":"65b009f5-0030-4c83-853c-cd8f56060540","Type":"ContainerDied","Data":"40cd027dd16e8fc265dc39c382ecdd64211a1e50ef0dc745696e5b1b8b188964"}
Nov 03 17:39:52.669055 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:52.669023    5838 generic.go:356] "Generic (PLEG): container finished" podID="bc84aa91-c637-4481-a958-9598a0f4913b" containerID="976c386af5ac3abc43a13ce18bfd2e8cf13bfb74f17b004c97c1164b306d81c6" exitCode=0
Nov 03 17:39:52.669195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:52.669068    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" event={"ID":"bc84aa91-c637-4481-a958-9598a0f4913b","Type":"ContainerDied","Data":"976c386af5ac3abc43a13ce18bfd2e8cf13bfb74f17b004c97c1164b306d81c6"}
Nov 03 17:39:53.680525 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:53.680387    5838 generic.go:356] "Generic (PLEG): container finished" podID="bc84aa91-c637-4481-a958-9598a0f4913b" containerID="508fa8be65241a97bb84f27e3111d3c86c9de39cc82f8afd1a32d3e63d9a30e9" exitCode=0
Nov 03 17:39:53.681170 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:53.680532    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" event={"ID":"bc84aa91-c637-4481-a958-9598a0f4913b","Type":"ContainerDied","Data":"508fa8be65241a97bb84f27e3111d3c86c9de39cc82f8afd1a32d3e63d9a30e9"}
Nov 03 17:39:53.689577 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:53.689539    5838 generic.go:356] "Generic (PLEG): container finished" podID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerID="76c953e71e24b350c9b8f8b5ca6ee81ba3c9776e6264dc4af958ce0841a152fd" exitCode=0
Nov 03 17:39:53.689708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:53.689600    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" event={"ID":"55ca8273-9d83-4643-8e42-860d846dfd7d","Type":"ContainerDied","Data":"76c953e71e24b350c9b8f8b5ca6ee81ba3c9776e6264dc4af958ce0841a152fd"}
Nov 03 17:39:54.718861 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:54.718779    5838 generic.go:356] "Generic (PLEG): container finished" podID="5aeab935-b169-4f94-988f-58928f6fd468" containerID="5e51aedd5a91ef6140a06ef0445cdbbac6bd3e2544df600e90a3376f9f20ff8a" exitCode=0
Nov 03 17:39:54.720162 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:54.719553    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r" event={"ID":"5aeab935-b169-4f94-988f-58928f6fd468","Type":"ContainerDied","Data":"5e51aedd5a91ef6140a06ef0445cdbbac6bd3e2544df600e90a3376f9f20ff8a"}
Nov 03 17:39:55.679455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:55.678667    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:39:55.679455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:55.678778    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:39:55.679455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:55.678817    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:39:55.725388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:55.724830    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:39:55.725388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:55.724937    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:39:55.725388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:55.724967    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:39:56.764562 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:56.763595    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx" event={"ID":"7cad14b8-057f-416d-938c-505f90358b9a","Type":"ContainerStarted","Data":"f62b9f5602755e22cebf751273837c0417cd2d4ab8cfd7ec9e43631e70bc48c5"}
Nov 03 17:39:56.772665 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:56.771893    5838 generic.go:356] "Generic (PLEG): container finished" podID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerID="dadaf9db96ca08249cc3b20a893621d708a244b9ea409d7cb70201e751edc70d" exitCode=0
Nov 03 17:39:56.772665 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:56.772603    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" event={"ID":"6ef56dac-91fe-41d4-b5be-db45b5844268","Type":"ContainerDied","Data":"dadaf9db96ca08249cc3b20a893621d708a244b9ea409d7cb70201e751edc70d"}
Nov 03 17:39:56.791175 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:56.790353    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" event={"ID":"3f4f6054-05f5-41c5-848f-b245cbff25d3","Type":"ContainerStarted","Data":"cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"}
Nov 03 17:39:56.941578 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:56.941025    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx" podStartSLOduration=2.699402831 podStartE2EDuration="11.941005838s" totalImagesPullingTime="9.241603007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:56.939481444 +0000 UTC m=+332.950219539" watchObservedRunningTime="2026-11-03 17:39:56.941005838 +0000 UTC m=+332.951743914"
Nov 03 17:39:56.996490 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:56.996454    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:57.635567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:57.633219    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:57.641275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:57.640983    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:57.769052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:57.768482    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podStartSLOduration=3.260799211 podStartE2EDuration="12.768459787s" totalImagesPullingTime="9.507660576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:57.188046929 +0000 UTC m=+333.198785003" watchObservedRunningTime="2026-11-03 17:39:57.768459787 +0000 UTC m=+333.779197859"
Nov 03 17:39:57.806055 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:57.804918    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:57.806746 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:57.806495    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:39:58.003677 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:58.003472    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:58.826722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:58.825824    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:58.827771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:58.827516    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx"
Nov 03 17:39:59.868076 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:39:59.868038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:00.026656 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:00.026620    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:06.920004 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:06.919962    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerStarted","Data":"d053ea65bcf0e2b41d127e237745575ff8085bd942b2f57d4aae7a416a63e323"}
Nov 03 17:40:07.043862 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:07.043786    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" podStartSLOduration=4.365872103 podStartE2EDuration="35.043766274s" totalImagesPullingTime="30.677894171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:07.042883959 +0000 UTC m=+343.053622055" watchObservedRunningTime="2026-11-03 17:40:07.043766274 +0000 UTC m=+343.054504345"
Nov 03 17:40:07.932323 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:07.932238    5838 generic.go:356] "Generic (PLEG): container finished" podID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerID="d053ea65bcf0e2b41d127e237745575ff8085bd942b2f57d4aae7a416a63e323" exitCode=0
Nov 03 17:40:07.933395 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:07.932846    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerDied","Data":"d053ea65bcf0e2b41d127e237745575ff8085bd942b2f57d4aae7a416a63e323"}
Nov 03 17:40:07.938685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:07.938378    5838 generic.go:356] "Generic (PLEG): container finished" podID="5aeab935-b169-4f94-988f-58928f6fd468" containerID="01b88f44ef2d97bece15da42395a3d4bdd34c2a60abdae6375ef5af9dd6eceaf" exitCode=0
Nov 03 17:40:07.938685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:07.938480    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r" event={"ID":"5aeab935-b169-4f94-988f-58928f6fd468","Type":"ContainerDied","Data":"01b88f44ef2d97bece15da42395a3d4bdd34c2a60abdae6375ef5af9dd6eceaf"}
Nov 03 17:40:08.948871 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:08.948805    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" event={"ID":"6ef56dac-91fe-41d4-b5be-db45b5844268","Type":"ContainerStarted","Data":"6e3d8a726ad9e205d056e5dd15a2ec73c79a5ba271d22e1abd2426fdfe48b8f9"}
Nov 03 17:40:09.108251 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.105687    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" podStartSLOduration=3.518148333 podStartE2EDuration="30.105672559s" totalImagesPullingTime="26.587524226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:09.104995084 +0000 UTC m=+345.115733178" watchObservedRunningTime="2026-11-03 17:40:09.105672559 +0000 UTC m=+345.116410645"
Nov 03 17:40:09.194501 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.194145    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:40:09.207815 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.207782    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:40:09.305467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.304316    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr\" (UniqueName: \"kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr\") pod \"5aeab935-b169-4f94-988f-58928f6fd468\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") "
Nov 03 17:40:09.305467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.304394    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle\") pod \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") "
Nov 03 17:40:09.305467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.304439    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9\" (UniqueName: \"kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9\") pod \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") "
Nov 03 17:40:09.305467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.304472    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util\") pod \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\" (UID: \"00ba4520-12ef-4c93-8d7c-0ee9960a3ada\") "
Nov 03 17:40:09.305467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.304582    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util\") pod \"5aeab935-b169-4f94-988f-58928f6fd468\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") "
Nov 03 17:40:09.305467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.304632    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle\") pod \"5aeab935-b169-4f94-988f-58928f6fd468\" (UID: \"5aeab935-b169-4f94-988f-58928f6fd468\") "
Nov 03 17:40:09.306378 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.306274    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle" pod "00ba4520-12ef-4c93-8d7c-0ee9960a3ada" (UID: "00ba4520-12ef-4c93-8d7c-0ee9960a3ada"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:09.306837 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.306720    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle" pod "5aeab935-b169-4f94-988f-58928f6fd468" (UID: "5aeab935-b169-4f94-988f-58928f6fd468"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:09.309404 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.309342    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr" pod "5aeab935-b169-4f94-988f-58928f6fd468" (UID: "5aeab935-b169-4f94-988f-58928f6fd468"). InnerVolumeSpecName "kube-api-access-w6qxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:09.309513 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.309473    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9" pod "00ba4520-12ef-4c93-8d7c-0ee9960a3ada" (UID: "00ba4520-12ef-4c93-8d7c-0ee9960a3ada"). InnerVolumeSpecName "kube-api-access-27gv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:09.316607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.316569    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util" pod "00ba4520-12ef-4c93-8d7c-0ee9960a3ada" (UID: "00ba4520-12ef-4c93-8d7c-0ee9960a3ada"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:09.316880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.316654    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util" pod "5aeab935-b169-4f94-988f-58928f6fd468" (UID: "5aeab935-b169-4f94-988f-58928f6fd468"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:09.406462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.406322    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.406462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.406367    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aeab935-b169-4f94-988f-58928f6fd468-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.406462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.406383    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qxr\" (UniqueName: \"kubernetes.io/projected/5aeab935-b169-4f94-988f-58928f6fd468-kube-api-access-w6qxr\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.406462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.406397    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.406462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.406413    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27gv9\" (UniqueName: \"kubernetes.io/projected/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-kube-api-access-27gv9\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.406462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.406427    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00ba4520-12ef-4c93-8d7c-0ee9960a3ada-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.961934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.961888    5838 generic.go:356] "Generic (PLEG): container finished" podID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerID="6e3d8a726ad9e205d056e5dd15a2ec73c79a5ba271d22e1abd2426fdfe48b8f9" exitCode=0
Nov 03 17:40:09.962406 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.961999    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" event={"ID":"6ef56dac-91fe-41d4-b5be-db45b5844268","Type":"ContainerDied","Data":"6e3d8a726ad9e205d056e5dd15a2ec73c79a5ba271d22e1abd2426fdfe48b8f9"}
Nov 03 17:40:09.969286 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.969252    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h"
Nov 03 17:40:09.969463 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.969253    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgq5h" event={"ID":"00ba4520-12ef-4c93-8d7c-0ee9960a3ada","Type":"ContainerDied","Data":"a94ac969d6a3a11941c7af298d2c157f810dceaa8a946140d20097adcb35f955"}
Nov 03 17:40:09.969463 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.969406    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94ac969d6a3a11941c7af298d2c157f810dceaa8a946140d20097adcb35f955"
Nov 03 17:40:09.978318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.978292    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r" event={"ID":"5aeab935-b169-4f94-988f-58928f6fd468","Type":"ContainerDied","Data":"3282b98ca5208d9d38f52b85ae3058876d64f939b768fb9219132fa41507d138"}
Nov 03 17:40:09.978583 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.978563    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3282b98ca5208d9d38f52b85ae3058876d64f939b768fb9219132fa41507d138"
Nov 03 17:40:09.978805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:09.978790    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tz82r"
Nov 03 17:40:10.368819 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:40:10.368779    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:40:10.987287 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:10.987078    5838 generic.go:356] "Generic (PLEG): container finished" podID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerID="29b25bbb0f83a846308fbed8f6b9ac5156a2a5ded07eced149e74c47846eefac" exitCode=0
Nov 03 17:40:10.988041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:10.987790    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9" event={"ID":"93729b8a-ded8-4ddc-ae53-da8267fd201e","Type":"ContainerDied","Data":"29b25bbb0f83a846308fbed8f6b9ac5156a2a5ded07eced149e74c47846eefac"}
Nov 03 17:40:10.996929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:10.996692    5838 generic.go:356] "Generic (PLEG): container finished" podID="56fc86bb-10c1-428c-9829-256bfea6f746" containerID="5cb460f161c8e68d0ded7291f17f01c70fa8601ea6f339fd0d8895550f6a36d7" exitCode=0
Nov 03 17:40:10.996929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:10.996801    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk" event={"ID":"56fc86bb-10c1-428c-9829-256bfea6f746","Type":"ContainerDied","Data":"5cb460f161c8e68d0ded7291f17f01c70fa8601ea6f339fd0d8895550f6a36d7"}
Nov 03 17:40:11.176279 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.176249    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:40:11.326978 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.326937    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2\" (UniqueName: \"kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2\") pod \"6ef56dac-91fe-41d4-b5be-db45b5844268\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") "
Nov 03 17:40:11.327210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.327062    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle\") pod \"6ef56dac-91fe-41d4-b5be-db45b5844268\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") "
Nov 03 17:40:11.327210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.327141    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util\") pod \"6ef56dac-91fe-41d4-b5be-db45b5844268\" (UID: \"6ef56dac-91fe-41d4-b5be-db45b5844268\") "
Nov 03 17:40:11.327668 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.327639    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle" pod "6ef56dac-91fe-41d4-b5be-db45b5844268" (UID: "6ef56dac-91fe-41d4-b5be-db45b5844268"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:11.329906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.329879    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2" pod "6ef56dac-91fe-41d4-b5be-db45b5844268" (UID: "6ef56dac-91fe-41d4-b5be-db45b5844268"). InnerVolumeSpecName "kube-api-access-9zkw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:11.332982 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.332954    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util" pod "6ef56dac-91fe-41d4-b5be-db45b5844268" (UID: "6ef56dac-91fe-41d4-b5be-db45b5844268"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:11.428688 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.428648    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:11.428688 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.428687    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef56dac-91fe-41d4-b5be-db45b5844268-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:11.428934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:11.428702    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zkw2\" (UniqueName: \"kubernetes.io/projected/6ef56dac-91fe-41d4-b5be-db45b5844268-kube-api-access-9zkw2\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.004544 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.004510    5838 generic.go:356] "Generic (PLEG): container finished" podID="65b009f5-0030-4c83-853c-cd8f56060540" containerID="93c9113010ef976d3070916a96b95fa37802d0e0faa29abc9eae728305c18e28" exitCode=0
Nov 03 17:40:12.005023 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.004604    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp" event={"ID":"65b009f5-0030-4c83-853c-cd8f56060540","Type":"ContainerDied","Data":"93c9113010ef976d3070916a96b95fa37802d0e0faa29abc9eae728305c18e28"}
Nov 03 17:40:12.006962 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.006939    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" event={"ID":"bc84aa91-c637-4481-a958-9598a0f4913b","Type":"ContainerStarted","Data":"d2c365a953993d37b6950aece62a7f91aabc58112329bbdb1d8c2edc771c0d1c"}
Nov 03 17:40:12.008954 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.008927    5838 generic.go:356] "Generic (PLEG): container finished" podID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerID="72fc9f42a3b7da14e4d98bd58e17ac638e6295638da0bd77258aa6371fe4fb04" exitCode=0
Nov 03 17:40:12.009086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.008988    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng" event={"ID":"fe82289f-0ef4-4ea9-b918-c2db1034ae34","Type":"ContainerDied","Data":"72fc9f42a3b7da14e4d98bd58e17ac638e6295638da0bd77258aa6371fe4fb04"}
Nov 03 17:40:12.010830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.010811    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b"
Nov 03 17:40:12.010925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.010812    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644v8s8b" event={"ID":"6ef56dac-91fe-41d4-b5be-db45b5844268","Type":"ContainerDied","Data":"b36de7c8259e80899d6c8fa860fcf12b7b47e1b93ad9bb68ac59648bf410e69a"}
Nov 03 17:40:12.010972 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.010931    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b36de7c8259e80899d6c8fa860fcf12b7b47e1b93ad9bb68ac59648bf410e69a"
Nov 03 17:40:12.012955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.012933    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" event={"ID":"55ca8273-9d83-4643-8e42-860d846dfd7d","Type":"ContainerStarted","Data":"461ea624dfa9056b0cb92dbb589d4f1154ddddf655f89b42430e375ccfb07ce8"}
Nov 03 17:40:12.178630 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.178605    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:40:12.181884 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.181865    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:40:12.337567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.337542    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util\") pod \"56fc86bb-10c1-428c-9829-256bfea6f746\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") "
Nov 03 17:40:12.337688 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.337583    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util\") pod \"93729b8a-ded8-4ddc-ae53-da8267fd201e\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") "
Nov 03 17:40:12.337688 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.337607    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle\") pod \"93729b8a-ded8-4ddc-ae53-da8267fd201e\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") "
Nov 03 17:40:12.337688 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.337639    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle\") pod \"56fc86bb-10c1-428c-9829-256bfea6f746\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") "
Nov 03 17:40:12.337846 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.337823    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89\" (UniqueName: \"kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89\") pod \"56fc86bb-10c1-428c-9829-256bfea6f746\" (UID: \"56fc86bb-10c1-428c-9829-256bfea6f746\") "
Nov 03 17:40:12.337901 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.337872    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx\" (UniqueName: \"kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx\") pod \"93729b8a-ded8-4ddc-ae53-da8267fd201e\" (UID: \"93729b8a-ded8-4ddc-ae53-da8267fd201e\") "
Nov 03 17:40:12.338255 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.338229    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle" pod "56fc86bb-10c1-428c-9829-256bfea6f746" (UID: "56fc86bb-10c1-428c-9829-256bfea6f746"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:12.338388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.338356    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle" pod "93729b8a-ded8-4ddc-ae53-da8267fd201e" (UID: "93729b8a-ded8-4ddc-ae53-da8267fd201e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:12.340061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.340034    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx" pod "93729b8a-ded8-4ddc-ae53-da8267fd201e" (UID: "93729b8a-ded8-4ddc-ae53-da8267fd201e"). InnerVolumeSpecName "kube-api-access-hcmnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:12.340386 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.340358    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89" pod "56fc86bb-10c1-428c-9829-256bfea6f746" (UID: "56fc86bb-10c1-428c-9829-256bfea6f746"). InnerVolumeSpecName "kube-api-access-ltp89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:12.343163 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.343140    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util" pod "56fc86bb-10c1-428c-9829-256bfea6f746" (UID: "56fc86bb-10c1-428c-9829-256bfea6f746"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:12.343312 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.343291    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util" pod "93729b8a-ded8-4ddc-ae53-da8267fd201e" (UID: "93729b8a-ded8-4ddc-ae53-da8267fd201e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:12.378752 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.378673    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" podStartSLOduration=3.361202595 podStartE2EDuration="36.378658551s" totalImagesPullingTime="33.017455956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:12.377369734 +0000 UTC m=+348.388107829" watchObservedRunningTime="2026-11-03 17:40:12.378658551 +0000 UTC m=+348.389396624"
Nov 03 17:40:12.439402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.439300    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmnx\" (UniqueName: \"kubernetes.io/projected/93729b8a-ded8-4ddc-ae53-da8267fd201e-kube-api-access-hcmnx\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.439402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.439334    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.439402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.439350    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.439402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.439362    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93729b8a-ded8-4ddc-ae53-da8267fd201e-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.439402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.439374    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56fc86bb-10c1-428c-9829-256bfea6f746-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.439402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.439384    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltp89\" (UniqueName: \"kubernetes.io/projected/56fc86bb-10c1-428c-9829-256bfea6f746-kube-api-access-ltp89\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:12.553463 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:12.553407    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" podStartSLOduration=3.295690603 podStartE2EDuration="32.55339146s" totalImagesPullingTime="29.257700857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:12.552949005 +0000 UTC m=+348.563687103" watchObservedRunningTime="2026-11-03 17:40:12.55339146 +0000 UTC m=+348.564129524"
Nov 03 17:40:13.020049 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.020016    5838 generic.go:356] "Generic (PLEG): container finished" podID="bc84aa91-c637-4481-a958-9598a0f4913b" containerID="d2c365a953993d37b6950aece62a7f91aabc58112329bbdb1d8c2edc771c0d1c" exitCode=0
Nov 03 17:40:13.020497 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.020093    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" event={"ID":"bc84aa91-c637-4481-a958-9598a0f4913b","Type":"ContainerDied","Data":"d2c365a953993d37b6950aece62a7f91aabc58112329bbdb1d8c2edc771c0d1c"}
Nov 03 17:40:13.022085 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.022065    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk"
Nov 03 17:40:13.022215 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.022083    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf2hfk" event={"ID":"56fc86bb-10c1-428c-9829-256bfea6f746","Type":"ContainerDied","Data":"2fe42f3e27620935e68b90bfc191529ae5b689e9150dc76636f85e4b5247fb81"}
Nov 03 17:40:13.022215 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.022117    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe42f3e27620935e68b90bfc191529ae5b689e9150dc76636f85e4b5247fb81"
Nov 03 17:40:13.024151 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.024131    5838 generic.go:356] "Generic (PLEG): container finished" podID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerID="461ea624dfa9056b0cb92dbb589d4f1154ddddf655f89b42430e375ccfb07ce8" exitCode=0
Nov 03 17:40:13.024254 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.024215    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" event={"ID":"55ca8273-9d83-4643-8e42-860d846dfd7d","Type":"ContainerDied","Data":"461ea624dfa9056b0cb92dbb589d4f1154ddddf655f89b42430e375ccfb07ce8"}
Nov 03 17:40:13.026075 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.026049    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9" event={"ID":"93729b8a-ded8-4ddc-ae53-da8267fd201e","Type":"ContainerDied","Data":"4c6130d474320ce3530f806a1de3af9d96b165080af12bc84c3119f95e91348f"}
Nov 03 17:40:13.026171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.026083    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8g6v9"
Nov 03 17:40:13.026171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.026099    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6130d474320ce3530f806a1de3af9d96b165080af12bc84c3119f95e91348f"
Nov 03 17:40:13.192077 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.192049    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:40:13.194467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.194447    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:40:13.264946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.264908    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle\") pod \"65b009f5-0030-4c83-853c-cd8f56060540\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") "
Nov 03 17:40:13.265140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.264974    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt\" (UniqueName: \"kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt\") pod \"65b009f5-0030-4c83-853c-cd8f56060540\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") "
Nov 03 17:40:13.265140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.265014    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle\") pod \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") "
Nov 03 17:40:13.265140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.265042    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util\") pod \"65b009f5-0030-4c83-853c-cd8f56060540\" (UID: \"65b009f5-0030-4c83-853c-cd8f56060540\") "
Nov 03 17:40:13.265140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.265078    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util\") pod \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") "
Nov 03 17:40:13.265337 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.265139    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2\" (UniqueName: \"kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2\") pod \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\" (UID: \"fe82289f-0ef4-4ea9-b918-c2db1034ae34\") "
Nov 03 17:40:13.265516 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.265469    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle" pod "65b009f5-0030-4c83-853c-cd8f56060540" (UID: "65b009f5-0030-4c83-853c-cd8f56060540"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:13.265645 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.265581    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle" pod "fe82289f-0ef4-4ea9-b918-c2db1034ae34" (UID: "fe82289f-0ef4-4ea9-b918-c2db1034ae34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:13.267390 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.267363    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt" pod "65b009f5-0030-4c83-853c-cd8f56060540" (UID: "65b009f5-0030-4c83-853c-cd8f56060540"). InnerVolumeSpecName "kube-api-access-k55rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:13.267725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.267679    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2" pod "fe82289f-0ef4-4ea9-b918-c2db1034ae34" (UID: "fe82289f-0ef4-4ea9-b918-c2db1034ae34"). InnerVolumeSpecName "kube-api-access-8nsb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:13.271426 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.271368    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util" pod "fe82289f-0ef4-4ea9-b918-c2db1034ae34" (UID: "fe82289f-0ef4-4ea9-b918-c2db1034ae34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:13.271975 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.271951    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util" pod "65b009f5-0030-4c83-853c-cd8f56060540" (UID: "65b009f5-0030-4c83-853c-cd8f56060540"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:13.366020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.365981    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:13.366020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.366011    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nsb2\" (UniqueName: \"kubernetes.io/projected/fe82289f-0ef4-4ea9-b918-c2db1034ae34-kube-api-access-8nsb2\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:13.366020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.366023    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:13.366251 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.366032    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55rt\" (UniqueName: \"kubernetes.io/projected/65b009f5-0030-4c83-853c-cd8f56060540-kube-api-access-k55rt\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:13.366251 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.366040    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe82289f-0ef4-4ea9-b918-c2db1034ae34-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:13.366251 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:13.366048    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b009f5-0030-4c83-853c-cd8f56060540-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:14.033143 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.033115    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp"
Nov 03 17:40:14.033617 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.033145    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c75jp" event={"ID":"65b009f5-0030-4c83-853c-cd8f56060540","Type":"ContainerDied","Data":"88cff690fdbe8383865fa25a130173b2d142c1e725f7b3f49d30c7838cbfd17a"}
Nov 03 17:40:14.033617 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.033233    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88cff690fdbe8383865fa25a130173b2d142c1e725f7b3f49d30c7838cbfd17a"
Nov 03 17:40:14.035322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.035298    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng"
Nov 03 17:40:14.035322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.035296    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72fbng" event={"ID":"fe82289f-0ef4-4ea9-b918-c2db1034ae34","Type":"ContainerDied","Data":"c6d33d972b01805ecb74666c94f0dd91b5be10993bb9aabb6938de654816e1eb"}
Nov 03 17:40:14.035496 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.035354    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d33d972b01805ecb74666c94f0dd91b5be10993bb9aabb6938de654816e1eb"
Nov 03 17:40:14.161263 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.161236    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:40:14.171251 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.171221    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle\") pod \"bc84aa91-c637-4481-a958-9598a0f4913b\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") "
Nov 03 17:40:14.171412 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.171305    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p\" (UniqueName: \"kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p\") pod \"bc84aa91-c637-4481-a958-9598a0f4913b\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") "
Nov 03 17:40:14.171412 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.171354    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util\") pod \"bc84aa91-c637-4481-a958-9598a0f4913b\" (UID: \"bc84aa91-c637-4481-a958-9598a0f4913b\") "
Nov 03 17:40:14.171854 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.171822    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle" pod "bc84aa91-c637-4481-a958-9598a0f4913b" (UID: "bc84aa91-c637-4481-a958-9598a0f4913b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:14.173624 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.173593    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p" pod "bc84aa91-c637-4481-a958-9598a0f4913b" (UID: "bc84aa91-c637-4481-a958-9598a0f4913b"). InnerVolumeSpecName "kube-api-access-h5b9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:14.177626 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.177600    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util" pod "bc84aa91-c637-4481-a958-9598a0f4913b" (UID: "bc84aa91-c637-4481-a958-9598a0f4913b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:14.193052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.193029    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:40:14.272407 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.272368    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle\") pod \"55ca8273-9d83-4643-8e42-860d846dfd7d\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") "
Nov 03 17:40:14.272603 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.272466    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util\") pod \"55ca8273-9d83-4643-8e42-860d846dfd7d\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") "
Nov 03 17:40:14.272603 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.272500    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5\" (UniqueName: \"kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5\") pod \"55ca8273-9d83-4643-8e42-860d846dfd7d\" (UID: \"55ca8273-9d83-4643-8e42-860d846dfd7d\") "
Nov 03 17:40:14.272818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.272800    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:14.272882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.272825    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5b9p\" (UniqueName: \"kubernetes.io/projected/bc84aa91-c637-4481-a958-9598a0f4913b-kube-api-access-h5b9p\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:14.272882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.272840    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc84aa91-c637-4481-a958-9598a0f4913b-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:14.273114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.273084    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle" pod "55ca8273-9d83-4643-8e42-860d846dfd7d" (UID: "55ca8273-9d83-4643-8e42-860d846dfd7d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:14.274836 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.274803    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5" pod "55ca8273-9d83-4643-8e42-860d846dfd7d" (UID: "55ca8273-9d83-4643-8e42-860d846dfd7d"). InnerVolumeSpecName "kube-api-access-shgq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:14.277704 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.277682    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util" pod "55ca8273-9d83-4643-8e42-860d846dfd7d" (UID: "55ca8273-9d83-4643-8e42-860d846dfd7d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:14.373615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.373533    5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-util\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:14.373615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.373558    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgq5\" (UniqueName: \"kubernetes.io/projected/55ca8273-9d83-4643-8e42-860d846dfd7d-kube-api-access-shgq5\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:14.373615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:14.373568    5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ca8273-9d83-4643-8e42-860d846dfd7d-bundle\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:40:15.042948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.042920    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m"
Nov 03 17:40:15.042948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.042941    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712b58m" event={"ID":"bc84aa91-c637-4481-a958-9598a0f4913b","Type":"ContainerDied","Data":"46a868b2da3e2cec6226e4d52881228b40a96cc979bdf44f45351124f4e274a2"}
Nov 03 17:40:15.043455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.042992    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46a868b2da3e2cec6226e4d52881228b40a96cc979bdf44f45351124f4e274a2"
Nov 03 17:40:15.044925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.044896    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc" event={"ID":"55ca8273-9d83-4643-8e42-860d846dfd7d","Type":"ContainerDied","Data":"a543a20dd057f7e554a3f505717717046b9911d1ce48671b6b943ad59801b47b"}
Nov 03 17:40:15.044925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.044919    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx8mvc"
Nov 03 17:40:15.045066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.044946    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a543a20dd057f7e554a3f505717717046b9911d1ce48671b6b943ad59801b47b"
Nov 03 17:40:15.159518 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.159475    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:15.322818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:15.322715    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:16.465764 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.465715    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"]
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466062    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466073    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466084    5838 cpu_manager.go:414] "containerMap: removing container" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="extract"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466089    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="extract"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466097    5838 cpu_manager.go:414] "containerMap: removing container" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466102    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466112    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="util"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466117    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="util"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466125    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="util"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466129    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="util"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466136    5838 cpu_manager.go:414] "containerMap: removing container" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466141    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466146    5838 cpu_manager.go:414] "containerMap: removing container" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466151    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466158    5838 cpu_manager.go:414] "containerMap: removing container" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="util"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466163    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="util"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466169    5838 cpu_manager.go:414] "containerMap: removing container" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466174    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="pull"
Nov 03 17:40:16.466172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466179    5838 cpu_manager.go:414] "containerMap: removing container" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466184    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466193    5838 cpu_manager.go:414] "containerMap: removing container" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466197    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466205    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466208    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466218    5838 cpu_manager.go:414] "containerMap: removing container" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466222    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466228    5838 cpu_manager.go:414] "containerMap: removing container" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466231    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466238    5838 cpu_manager.go:414] "containerMap: removing container" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466242    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466248    5838 cpu_manager.go:414] "containerMap: removing container" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466251    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466257    5838 cpu_manager.go:414] "containerMap: removing container" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466261    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466269    5838 cpu_manager.go:414] "containerMap: removing container" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466273    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466278    5838 cpu_manager.go:414] "containerMap: removing container" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466282    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466287    5838 cpu_manager.go:414] "containerMap: removing container" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466292    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466299    5838 cpu_manager.go:414] "containerMap: removing container" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466303    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466308    5838 cpu_manager.go:414] "containerMap: removing container" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466312    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466317    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466321    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466327    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466331    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466340    5838 cpu_manager.go:414] "containerMap: removing container" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466345    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466350    5838 cpu_manager.go:414] "containerMap: removing container" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466354    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="pull"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466362    5838 cpu_manager.go:414] "containerMap: removing container" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466366    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="util"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466415    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="93729b8a-ded8-4ddc-ae53-da8267fd201e" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466425    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b009f5-0030-4c83-853c-cd8f56060540" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466432    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ba4520-12ef-4c93-8d7c-0ee9960a3ada" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466438    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe82289f-0ef4-4ea9-b918-c2db1034ae34" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466443    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc84aa91-c637-4481-a958-9598a0f4913b" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466449    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ca8273-9d83-4643-8e42-860d846dfd7d" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466453    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aeab935-b169-4f94-988f-58928f6fd468" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466458    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef56dac-91fe-41d4-b5be-db45b5844268" containerName="extract"
Nov 03 17:40:16.466694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.466465    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fc86bb-10c1-428c-9829-256bfea6f746" containerName="extract"
Nov 03 17:40:16.488646 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.488612    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.531798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.531759    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"]
Nov 03 17:40:16.580307 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.580265    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 17:40:16.580307 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.580271    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:40:16.580528 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.580361    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-qptw4\""
Nov 03 17:40:16.591789 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.591757    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebd338b9-6245-405f-b260-02b7fd2c0cc6-tmp\") pod \"jobset-operator-674cd59dc5-pldg9\" (UID: \"ebd338b9-6245-405f-b260-02b7fd2c0cc6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.591846 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.591831    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/ebd338b9-6245-405f-b260-02b7fd2c0cc6-kube-api-access-hvp6c\") pod \"jobset-operator-674cd59dc5-pldg9\" (UID: \"ebd338b9-6245-405f-b260-02b7fd2c0cc6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.692773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.692709    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebd338b9-6245-405f-b260-02b7fd2c0cc6-tmp\") pod \"jobset-operator-674cd59dc5-pldg9\" (UID: \"ebd338b9-6245-405f-b260-02b7fd2c0cc6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.692927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.692802    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/ebd338b9-6245-405f-b260-02b7fd2c0cc6-kube-api-access-hvp6c\") pod \"jobset-operator-674cd59dc5-pldg9\" (UID: \"ebd338b9-6245-405f-b260-02b7fd2c0cc6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.693135 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.693114    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebd338b9-6245-405f-b260-02b7fd2c0cc6-tmp\") pod \"jobset-operator-674cd59dc5-pldg9\" (UID: \"ebd338b9-6245-405f-b260-02b7fd2c0cc6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.758958 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.758881    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/ebd338b9-6245-405f-b260-02b7fd2c0cc6-kube-api-access-hvp6c\") pod \"jobset-operator-674cd59dc5-pldg9\" (UID: \"ebd338b9-6245-405f-b260-02b7fd2c0cc6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:16.829214 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:16.829174    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"
Nov 03 17:40:17.237789 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:17.237604    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9"]
Nov 03 17:40:17.240474 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:17.240438    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd338b9_6245_405f_b260_02b7fd2c0cc6.slice/crio-8af625f7b5c8bdb013bc47c2b1d62cc9dbd4ddfc87a283d4017a08e6cbed1b0d WatchSource:0}: Error finding container 8af625f7b5c8bdb013bc47c2b1d62cc9dbd4ddfc87a283d4017a08e6cbed1b0d: Status 404 returned error can't find the container with id 8af625f7b5c8bdb013bc47c2b1d62cc9dbd4ddfc87a283d4017a08e6cbed1b0d
Nov 03 17:40:17.242052 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:17.242029    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:40:18.065873 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:18.065837    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9" event={"ID":"ebd338b9-6245-405f-b260-02b7fd2c0cc6","Type":"ContainerStarted","Data":"8af625f7b5c8bdb013bc47c2b1d62cc9dbd4ddfc87a283d4017a08e6cbed1b0d"}
Nov 03 17:40:22.087291 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:22.087243    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9" event={"ID":"ebd338b9-6245-405f-b260-02b7fd2c0cc6","Type":"ContainerStarted","Data":"323731e898aaab52eeef4c5eb99538f8e259ebfccb251a350671df3e4ecf84b2"}
Nov 03 17:40:22.209614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:22.209560    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pldg9" podStartSLOduration=2.213810186 podStartE2EDuration="6.2095446s" totalImagesPullingTime="3.995734414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:22.208121229 +0000 UTC m=+358.218859324" watchObservedRunningTime="2026-11-03 17:40:22.2095446 +0000 UTC m=+358.220282672"
Nov 03 17:40:22.266686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:22.266647    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:22.426380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:22.426343    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:23.483153 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.483108    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:23.735586 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.735484    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"]
Nov 03 17:40:23.738058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.738033    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.744192 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.744163    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.744318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.744216    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.744370 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.744334    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.744427 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.744411    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.744462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.744436    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zccq7\" (UniqueName: \"kubernetes.io/projected/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-kube-api-access-zccq7\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.744514 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.744498    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.793128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.793095    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 17:40:23.793128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.793100    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 17:40:23.793349 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.793146    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 17:40:23.802231 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.802195    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:23.842565 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.842528    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kqpc6\""
Nov 03 17:40:23.842801 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.842783    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 17:40:23.842883 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.842805    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 17:40:23.845228 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845207    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.845281 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845235    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zccq7\" (UniqueName: \"kubernetes.io/projected/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-kube-api-access-zccq7\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.845281 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845270    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.845369 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845309    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.845369 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845331    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.845369 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845356    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.845710 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.845685    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.847914 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.847882    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.848038 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.848007    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.848141 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.848113    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.848201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.848163    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.858655 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.858628    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"]
Nov 03 17:40:23.916930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.916894    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zccq7\" (UniqueName: \"kubernetes.io/projected/3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5-kube-api-access-zccq7\") pod \"opendatahub-operator-controller-manager-57dbb86d66-hhf88\" (UID: \"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:23.977015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:23.976970    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:24.051230 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:24.051152    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:24.147522 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:24.147487    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:24.319076 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:24.318997    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:24.550148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:24.550099    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"]
Nov 03 17:40:24.552532 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:24.552491    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c660dd4_f8f9_4013_ae1b_9bbe0f0b48a5.slice/crio-71b44e5a8acca0d10e45dd6a433f37be504d40f19cfd0ead78818398c58c36f4 WatchSource:0}: Error finding container 71b44e5a8acca0d10e45dd6a433f37be504d40f19cfd0ead78818398c58c36f4: Status 404 returned error can't find the container with id 71b44e5a8acca0d10e45dd6a433f37be504d40f19cfd0ead78818398c58c36f4
Nov 03 17:40:25.102060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:25.102022    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88" event={"ID":"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5","Type":"ContainerStarted","Data":"71b44e5a8acca0d10e45dd6a433f37be504d40f19cfd0ead78818398c58c36f4"}
Nov 03 17:40:26.049340 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:26.049299    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:27.067326 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:27.067280    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:28.118619 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:28.118585    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88" event={"ID":"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5","Type":"ContainerStarted","Data":"bba1420af60ea434a2ef5b14d4c62d29f79bb72f348e8ea4c069c732d00596e7"}
Nov 03 17:40:29.130361 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.130324    5838 generic.go:356] "Generic (PLEG): container finished" podID="3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5" containerID="bba1420af60ea434a2ef5b14d4c62d29f79bb72f348e8ea4c069c732d00596e7" exitCode=0
Nov 03 17:40:29.130806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.130414    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88" event={"ID":"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5","Type":"ContainerDied","Data":"bba1420af60ea434a2ef5b14d4c62d29f79bb72f348e8ea4c069c732d00596e7"}
Nov 03 17:40:29.384831 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.384749    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"]
Nov 03 17:40:29.387310 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.387292    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.442822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.442785    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 17:40:29.442822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.442786    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8hrbm\""
Nov 03 17:40:29.442822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.442790    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:40:29.454666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.454641    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"]
Nov 03 17:40:29.491748 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.491689    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8751411-9e57-49d0-b4a2-b6008d7f90d2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-l54ns\" (UID: \"d8751411-9e57-49d0-b4a2-b6008d7f90d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.491921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.491811    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx45g\" (UniqueName: \"kubernetes.io/projected/d8751411-9e57-49d0-b4a2-b6008d7f90d2-kube-api-access-mx45g\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-l54ns\" (UID: \"d8751411-9e57-49d0-b4a2-b6008d7f90d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.592293 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.592253    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8751411-9e57-49d0-b4a2-b6008d7f90d2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-l54ns\" (UID: \"d8751411-9e57-49d0-b4a2-b6008d7f90d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.592293 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.592296    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx45g\" (UniqueName: \"kubernetes.io/projected/d8751411-9e57-49d0-b4a2-b6008d7f90d2-kube-api-access-mx45g\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-l54ns\" (UID: \"d8751411-9e57-49d0-b4a2-b6008d7f90d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.592760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.592704    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8751411-9e57-49d0-b4a2-b6008d7f90d2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-l54ns\" (UID: \"d8751411-9e57-49d0-b4a2-b6008d7f90d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.660900 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.660866    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx45g\" (UniqueName: \"kubernetes.io/projected/d8751411-9e57-49d0-b4a2-b6008d7f90d2-kube-api-access-mx45g\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-l54ns\" (UID: \"d8751411-9e57-49d0-b4a2-b6008d7f90d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:29.697984 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:29.697943    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"
Nov 03 17:40:30.136712 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:30.136680    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns"]
Nov 03 17:40:30.139663 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:30.139632    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88" event={"ID":"3c660dd4-f8f9-4013-ae1b-9bbe0f0b48a5","Type":"ContainerStarted","Data":"853952dc0eba51e32fd38571348b78ac1da3d8ede103c6e0e8aa0cc44f0fb7b4"}
Nov 03 17:40:30.139812 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:30.139774    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:30.140462 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:30.140438    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8751411_9e57_49d0_b4a2_b6008d7f90d2.slice/crio-1b6825e369d04021c6b30b71ef3f30d3e0cde501a1fbc405a8a9e73e4e540f2c WatchSource:0}: Error finding container 1b6825e369d04021c6b30b71ef3f30d3e0cde501a1fbc405a8a9e73e4e540f2c: Status 404 returned error can't find the container with id 1b6825e369d04021c6b30b71ef3f30d3e0cde501a1fbc405a8a9e73e4e540f2c
Nov 03 17:40:30.287342 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:30.287223    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88" podStartSLOduration=2.598955162 podStartE2EDuration="7.287208041s" totalImagesPullingTime="4.28230985s" totalInitContainerRuntime="405.943029ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:30.285177283 +0000 UTC m=+366.295915353" watchObservedRunningTime="2026-11-03 17:40:30.287208041 +0000 UTC m=+366.297946167"
Nov 03 17:40:31.146308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:31.146267    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns" event={"ID":"d8751411-9e57-49d0-b4a2-b6008d7f90d2","Type":"ContainerStarted","Data":"1b6825e369d04021c6b30b71ef3f30d3e0cde501a1fbc405a8a9e73e4e540f2c"}
Nov 03 17:40:32.257883 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:32.257846    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:32.416524 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:32.416483    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:33.890380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:33.890343    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:34.165128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:34.165038    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns" event={"ID":"d8751411-9e57-49d0-b4a2-b6008d7f90d2","Type":"ContainerStarted","Data":"0c299feebef81f9809cafd1840ec8172def0c3fb75db0d7ade3890253a2df1da"}
Nov 03 17:40:34.371354 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:34.371298    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-l54ns" podStartSLOduration=2.386908675 podStartE2EDuration="5.371282585s" totalImagesPullingTime="2.98437391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:34.346319603 +0000 UTC m=+370.357057690" watchObservedRunningTime="2026-11-03 17:40:34.371282585 +0000 UTC m=+370.382020658"
Nov 03 17:40:35.618168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.618132    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"]
Nov 03 17:40:35.622892 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.622848    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:35.646671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.646637    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944q6\" (UniqueName: \"kubernetes.io/projected/146565e0-81b4-4c59-8505-b8f99bcfc8eb-kube-api-access-944q6\") pod \"dns-operator-controller-manager-8557cbdb84-9l9z8\" (UID: \"146565e0-81b4-4c59-8505-b8f99bcfc8eb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:35.680531 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.680501    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 17:40:35.680531 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.680524    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 17:40:35.730146 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.730113    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 17:40:35.730316 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.730276    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qggbs\""
Nov 03 17:40:35.731862 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.731826    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"]
Nov 03 17:40:35.747676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.747645    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944q6\" (UniqueName: \"kubernetes.io/projected/146565e0-81b4-4c59-8505-b8f99bcfc8eb-kube-api-access-944q6\") pod \"dns-operator-controller-manager-8557cbdb84-9l9z8\" (UID: \"146565e0-81b4-4c59-8505-b8f99bcfc8eb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:35.811451 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.811422    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944q6\" (UniqueName: \"kubernetes.io/projected/146565e0-81b4-4c59-8505-b8f99bcfc8eb-kube-api-access-944q6\") pod \"dns-operator-controller-manager-8557cbdb84-9l9z8\" (UID: \"146565e0-81b4-4c59-8505-b8f99bcfc8eb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:35.937517 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.937483    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:35.967098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:35.967066    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:36.143874 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.143834    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:36.324541 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.324503    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"]
Nov 03 17:40:36.326952 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:36.326913    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146565e0_81b4_4c59_8505_b8f99bcfc8eb.slice/crio-a2a0d603f589ce5643168d043df9f454b9b0d11f539c9f9cf93e847a0e436e2d WatchSource:0}: Error finding container a2a0d603f589ce5643168d043df9f454b9b0d11f539c9f9cf93e847a0e436e2d: Status 404 returned error can't find the container with id a2a0d603f589ce5643168d043df9f454b9b0d11f539c9f9cf93e847a0e436e2d
Nov 03 17:40:36.399823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.399783    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:36.611488 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.611389    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:36.745540 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.745509    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"]
Nov 03 17:40:36.750235 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.750213    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:36.783595 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.783561    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:36.812595 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.812565    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 17:40:36.812806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.812638    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 17:40:36.813830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.813811    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jrg7f\""
Nov 03 17:40:36.814188 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.814171    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"]
Nov 03 17:40:36.857345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.857284    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkkn\" (UniqueName: \"kubernetes.io/projected/16e0e47b-4ef8-471c-9b09-19086501f5e6-kube-api-access-znkkn\") pod \"cert-manager-webhook-5dbbbf75cc-xr7pm\" (UID: \"16e0e47b-4ef8-471c-9b09-19086501f5e6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:36.857505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.857438    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16e0e47b-4ef8-471c-9b09-19086501f5e6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xr7pm\" (UID: \"16e0e47b-4ef8-471c-9b09-19086501f5e6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:36.958680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.958646    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16e0e47b-4ef8-471c-9b09-19086501f5e6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xr7pm\" (UID: \"16e0e47b-4ef8-471c-9b09-19086501f5e6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:36.958881 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:36.958709    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkkn\" (UniqueName: \"kubernetes.io/projected/16e0e47b-4ef8-471c-9b09-19086501f5e6-kube-api-access-znkkn\") pod \"cert-manager-webhook-5dbbbf75cc-xr7pm\" (UID: \"16e0e47b-4ef8-471c-9b09-19086501f5e6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:37.030053 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:37.030018    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16e0e47b-4ef8-471c-9b09-19086501f5e6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xr7pm\" (UID: \"16e0e47b-4ef8-471c-9b09-19086501f5e6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:37.030207 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:37.030147    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkkn\" (UniqueName: \"kubernetes.io/projected/16e0e47b-4ef8-471c-9b09-19086501f5e6-kube-api-access-znkkn\") pod \"cert-manager-webhook-5dbbbf75cc-xr7pm\" (UID: \"16e0e47b-4ef8-471c-9b09-19086501f5e6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:37.063102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:37.063066    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:37.180889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:37.180852    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8" event={"ID":"146565e0-81b4-4c59-8505-b8f99bcfc8eb","Type":"ContainerStarted","Data":"a2a0d603f589ce5643168d043df9f454b9b0d11f539c9f9cf93e847a0e436e2d"}
Nov 03 17:40:37.447640 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:37.447615    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"]
Nov 03 17:40:37.449794 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:37.449767    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16e0e47b_4ef8_471c_9b09_19086501f5e6.slice/crio-fcd58615c8657f86f480627cfd42c53ca3c0a24d65f5aee737c34ab75be57d4b WatchSource:0}: Error finding container fcd58615c8657f86f480627cfd42c53ca3c0a24d65f5aee737c34ab75be57d4b: Status 404 returned error can't find the container with id fcd58615c8657f86f480627cfd42c53ca3c0a24d65f5aee737c34ab75be57d4b
Nov 03 17:40:38.188752 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:38.188699    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm" event={"ID":"16e0e47b-4ef8-471c-9b09-19086501f5e6","Type":"ContainerStarted","Data":"fcd58615c8657f86f480627cfd42c53ca3c0a24d65f5aee737c34ab75be57d4b"}
Nov 03 17:40:38.409146 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:38.409104    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:38.844106 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:38.844073    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:39.454870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.454838    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"]
Nov 03 17:40:39.465213 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.465183    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:39.521484 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.521446    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"]
Nov 03 17:40:39.583054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.583021    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dkfrn\""
Nov 03 17:40:39.583257 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.583133    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7gp\" (UniqueName: \"kubernetes.io/projected/29ba8591-1315-4eea-bf6a-1c81882666e7-kube-api-access-fv7gp\") pod \"limitador-operator-controller-manager-5778bb4449-rblzq\" (UID: \"29ba8591-1315-4eea-bf6a-1c81882666e7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:39.684658 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.684617    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7gp\" (UniqueName: \"kubernetes.io/projected/29ba8591-1315-4eea-bf6a-1c81882666e7-kube-api-access-fv7gp\") pod \"limitador-operator-controller-manager-5778bb4449-rblzq\" (UID: \"29ba8591-1315-4eea-bf6a-1c81882666e7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:39.751200 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.751129    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7gp\" (UniqueName: \"kubernetes.io/projected/29ba8591-1315-4eea-bf6a-1c81882666e7-kube-api-access-fv7gp\") pod \"limitador-operator-controller-manager-5778bb4449-rblzq\" (UID: \"29ba8591-1315-4eea-bf6a-1c81882666e7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:39.782102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.781913    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:39.830639 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:39.830593    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:40.179208 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:40.179037    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"]
Nov 03 17:40:40.181892 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:40.181863    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ba8591_1315_4eea_bf6a_1c81882666e7.slice/crio-989b9a815893edc5e20061a4186a1dab503520ebcd37df5f6aab739113819a25 WatchSource:0}: Error finding container 989b9a815893edc5e20061a4186a1dab503520ebcd37df5f6aab739113819a25: Status 404 returned error can't find the container with id 989b9a815893edc5e20061a4186a1dab503520ebcd37df5f6aab739113819a25
Nov 03 17:40:40.203395 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:40.203358    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq" event={"ID":"29ba8591-1315-4eea-bf6a-1c81882666e7","Type":"ContainerStarted","Data":"989b9a815893edc5e20061a4186a1dab503520ebcd37df5f6aab739113819a25"}
Nov 03 17:40:40.205595 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:40.205558    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8" event={"ID":"146565e0-81b4-4c59-8505-b8f99bcfc8eb","Type":"ContainerStarted","Data":"0846bb8766a76f5615abfef665058f60dd1e8f07da01fbfab7eabbd7d03fd5fe"}
Nov 03 17:40:40.206138 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:40.206101    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:40.330409 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:40.330298    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8" podStartSLOduration=1.6415410769999998 podStartE2EDuration="5.330277371s" totalImagesPullingTime="3.688736294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:40.327808854 +0000 UTC m=+376.338546946" watchObservedRunningTime="2026-11-03 17:40:40.330277371 +0000 UTC m=+376.341015445"
Nov 03 17:40:40.341625 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:40:40.341586    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:40:40.658073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:40.658032    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:41.148469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:41.148440    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-hhf88"
Nov 03 17:40:41.212930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:41.212893    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm" event={"ID":"16e0e47b-4ef8-471c-9b09-19086501f5e6","Type":"ContainerStarted","Data":"d3e8f30099d1bd0c4c15718fe2e253478ed452c9e1a27e6d8b1570d7e84bfeb4"}
Nov 03 17:40:41.213114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:41.212986    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:41.523031 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:41.522976    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm" podStartSLOduration=1.9391789959999999 podStartE2EDuration="5.522959552s" totalImagesPullingTime="3.583780556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:41.521320514 +0000 UTC m=+377.532058608" watchObservedRunningTime="2026-11-03 17:40:41.522959552 +0000 UTC m=+377.533697624"
Nov 03 17:40:41.632987 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:41.632921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:42.258429 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:42.258387    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:42.420798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:42.420764    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:43.225388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:43.225294    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq" event={"ID":"29ba8591-1315-4eea-bf6a-1c81882666e7","Type":"ContainerStarted","Data":"5dd64ff587118dd9ccd86ae63da923324f0b8d78766f4832aacefd81d9000e87"}
Nov 03 17:40:43.225544 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:43.225417    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:44.451112 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:44.451083    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:44.636950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:44.636902    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:46.273168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:46.273132    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:46.537589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:46.537491    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:46.847895 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:46.847802    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:47.219847 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:47.219817    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xr7pm"
Nov 03 17:40:47.340883 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:47.340835    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq" podStartSLOduration=5.567458447 podStartE2EDuration="8.340820068s" totalImagesPullingTime="2.773361621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:43.352518007 +0000 UTC m=+379.363256080" watchObservedRunningTime="2026-11-03 17:40:47.340820068 +0000 UTC m=+383.351558148"
Nov 03 17:40:47.834783 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:47.834749    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:48.133559 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:48.133454    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:48.347154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:48.347110    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:50.994712 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:50.994678    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"]
Nov 03 17:40:50.998691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:50.998663    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:40:51.061017 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.060986    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-d4kpg\""
Nov 03 17:40:51.098631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.098595    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xg2\" (UniqueName: \"kubernetes.io/projected/ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0-kube-api-access-95xg2\") pod \"authorino-operator-7b7bb4bc77-z8zbr\" (UID: \"ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:40:51.102575 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.102544    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:51.106856 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.106830    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"]
Nov 03 17:40:51.200084 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.200041    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xg2\" (UniqueName: \"kubernetes.io/projected/ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0-kube-api-access-95xg2\") pod \"authorino-operator-7b7bb4bc77-z8zbr\" (UID: \"ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:40:51.215003 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.214974    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9l9z8"
Nov 03 17:40:51.267576 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.267503    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xg2\" (UniqueName: \"kubernetes.io/projected/ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0-kube-api-access-95xg2\") pod \"authorino-operator-7b7bb4bc77-z8zbr\" (UID: \"ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:40:51.286971 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.286944    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:51.312169 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.312138    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:40:51.700143 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:51.700119    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"]
Nov 03 17:40:51.702706 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:51.702679    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded03bf6a_2b81_4466_ba6b_6bc624d1f3c0.slice/crio-663fc59587ae7f8e07b1e1eb95fe0e2feddc91256de4cb9e6113955f0a953beb WatchSource:0}: Error finding container 663fc59587ae7f8e07b1e1eb95fe0e2feddc91256de4cb9e6113955f0a953beb: Status 404 returned error can't find the container with id 663fc59587ae7f8e07b1e1eb95fe0e2feddc91256de4cb9e6113955f0a953beb
Nov 03 17:40:52.272152 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:52.272122    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr" event={"ID":"ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0","Type":"ContainerStarted","Data":"663fc59587ae7f8e07b1e1eb95fe0e2feddc91256de4cb9e6113955f0a953beb"}
Nov 03 17:40:53.261176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:53.261137    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:53.423881 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:53.423841    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:54.232290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:54.232259    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rblzq"
Nov 03 17:40:55.288134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:55.288032    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr" event={"ID":"ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0","Type":"ContainerStarted","Data":"00fd8b7d760d1f9e54b044ed9d8b40bb1286eef31ab5dcbb6690b24c916d5bc3"}
Nov 03 17:40:55.288596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:55.288141    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:40:55.410005 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:55.409951    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr" podStartSLOduration=2.094417468 podStartE2EDuration="5.409935982s" totalImagesPullingTime="3.315518514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:55.408856087 +0000 UTC m=+391.419594185" watchObservedRunningTime="2026-11-03 17:40:55.409935982 +0000 UTC m=+391.420674055"
Nov 03 17:40:56.114081 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:56.114047    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:56.276662 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:56.276626    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:57.293219 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.293183    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:57.327396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.327360    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"]
Nov 03 17:40:57.331073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.331050    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.385845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.385811    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 17:40:57.435845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.435810    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 17:40:57.435845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.435813    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pknc5\""
Nov 03 17:40:57.437948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.437926    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"]
Nov 03 17:40:57.461803 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.461771    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.461992 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.461960    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.462051 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.462007    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67qc\" (UniqueName: \"kubernetes.io/projected/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-kube-api-access-r67qc\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.505891 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.505849    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:57.563131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.563049    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.563131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.563094    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67qc\" (UniqueName: \"kubernetes.io/projected/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-kube-api-access-r67qc\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.563131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.563118    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.563771 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.563712    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.565725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.565703    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.627176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.627137    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67qc\" (UniqueName: \"kubernetes.io/projected/0bdb49a9-4db8-4775-adbf-cd1e1779dbb2-kube-api-access-r67qc\") pod \"kuadrant-console-plugin-66c4fb575d-76xkn\" (UID: \"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:57.644781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:57.644747    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"
Nov 03 17:40:58.024879 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:58.024850    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn"]
Nov 03 17:40:58.027289 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:40:58.027246    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bdb49a9_4db8_4775_adbf_cd1e1779dbb2.slice/crio-7398131d1cf43c3081a05f01b07dd3589b7ff6bb828fd222ce421a4ce2db48f3 WatchSource:0}: Error finding container 7398131d1cf43c3081a05f01b07dd3589b7ff6bb828fd222ce421a4ce2db48f3: Status 404 returned error can't find the container with id 7398131d1cf43c3081a05f01b07dd3589b7ff6bb828fd222ce421a4ce2db48f3
Nov 03 17:40:58.303832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:40:58.303709    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn" event={"ID":"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2","Type":"ContainerStarted","Data":"7398131d1cf43c3081a05f01b07dd3589b7ff6bb828fd222ce421a4ce2db48f3"}
Nov 03 17:41:01.269129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:01.269095    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:01.458496 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:01.458458    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:01.661752 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:01.661684    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:01.881803 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:01.881753    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:02.260906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:02.260849    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:03.706045 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.706003    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"]
Nov 03 17:41:03.709952 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.709924    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:03.726043 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.726011    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"]
Nov 03 17:41:03.774290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.774209    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-snz9x\""
Nov 03 17:41:03.826925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.826892    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v9vxr\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:03.827090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.827028    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4s8\" (UniqueName: \"kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8\") pod \"kuadrant-operator-controller-manager-678d87766f-v9vxr\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:03.901422 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.901384    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:03.927670 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.927637    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4s8\" (UniqueName: \"kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8\") pod \"kuadrant-operator-controller-manager-678d87766f-v9vxr\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:03.927880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.927703    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v9vxr\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:03.928122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.928104    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v9vxr\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:03.992570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:03.992539    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4s8\" (UniqueName: \"kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8\") pod \"kuadrant-operator-controller-manager-678d87766f-v9vxr\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:04.023582 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.023550    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:04.212348 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.212317    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:04.341076 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.340983    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn" event={"ID":"0bdb49a9-4db8-4775-adbf-cd1e1779dbb2","Type":"ContainerStarted","Data":"2c7184925f4de9dfce1184d33fcbed57fd554d328213ee7c1a599f5fc327ecdc"}
Nov 03 17:41:04.412804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.412761    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:04.457116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.457086    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"]
Nov 03 17:41:04.459700 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:41:04.459666    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac802f81_9b7f_4abe_9b66_be17114d4a7c.slice/crio-24e976d70a507d93e6c94a070b3d54b88886aa0f9cd3f2b6b7066e44bd61eeb7 WatchSource:0}: Error finding container 24e976d70a507d93e6c94a070b3d54b88886aa0f9cd3f2b6b7066e44bd61eeb7: Status 404 returned error can't find the container with id 24e976d70a507d93e6c94a070b3d54b88886aa0f9cd3f2b6b7066e44bd61eeb7
Nov 03 17:41:04.459856 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.459794    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-76xkn" podStartSLOduration=2.008187471 podStartE2EDuration="7.459778899s" totalImagesPullingTime="5.451591428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:04.458341292 +0000 UTC m=+400.469079364" watchObservedRunningTime="2026-11-03 17:41:04.459778899 +0000 UTC m=+400.470516971"
Nov 03 17:41:04.579478 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.579431    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"]
Nov 03 17:41:04.584134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.584106    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.651077 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.651047    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zbrfl\""
Nov 03 17:41:04.653002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.652972    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"]
Nov 03 17:41:04.705717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.705685    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:04.736379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736338    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736401    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736481    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78pt\" (UniqueName: \"kubernetes.io/projected/c815c4f2-540a-4951-910a-b75c9ad9a72d-kube-api-access-x78pt\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736517    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736599    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736641    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736675    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736716    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.736828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.736819    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c815c4f2-540a-4951-910a-b75c9ad9a72d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838105 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838067    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838105 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838112    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78pt\" (UniqueName: \"kubernetes.io/projected/c815c4f2-540a-4951-910a-b75c9ad9a72d-kube-api-access-x78pt\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838132    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838159    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838203    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838235    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838422    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838484    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c815c4f2-540a-4951-910a-b75c9ad9a72d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838522    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838700    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838775    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.838944 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.838921    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.839130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.839026    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.839212 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.839156    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c815c4f2-540a-4951-910a-b75c9ad9a72d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.840848 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.840818    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.841218 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.841192    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.885565 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.885436    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:04.902963 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.902922    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c815c4f2-540a-4951-910a-b75c9ad9a72d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:04.903209 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:04.903180    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78pt\" (UniqueName: \"kubernetes.io/projected/c815c4f2-540a-4951-910a-b75c9ad9a72d-kube-api-access-x78pt\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85rq9ds\" (UID: \"c815c4f2-540a-4951-910a-b75c9ad9a72d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:05.198153 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:05.198112    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:05.347387 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:05.347345    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr" event={"ID":"ac802f81-9b7f-4abe-9b66-be17114d4a7c","Type":"ContainerStarted","Data":"24e976d70a507d93e6c94a070b3d54b88886aa0f9cd3f2b6b7066e44bd61eeb7"}
Nov 03 17:41:05.639297 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:05.639270    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"]
Nov 03 17:41:05.640940 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:41:05.640913    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc815c4f2_540a_4951_910a_b75c9ad9a72d.slice/crio-56273d304634eb3f6eac2372a2df1535d9080cbf43892d7796f6c7e3cae16dcb WatchSource:0}: Error finding container 56273d304634eb3f6eac2372a2df1535d9080cbf43892d7796f6c7e3cae16dcb: Status 404 returned error can't find the container with id 56273d304634eb3f6eac2372a2df1535d9080cbf43892d7796f6c7e3cae16dcb
Nov 03 17:41:05.643183 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:05.643147    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:41:05.643286 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:05.643229    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:41:05.643286 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:05.643279    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:41:06.295333 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:06.295299    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z8zbr"
Nov 03 17:41:06.355988 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:06.355952    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds" event={"ID":"c815c4f2-540a-4951-910a-b75c9ad9a72d","Type":"ContainerStarted","Data":"ed266ae1159db37646e425c439993e75c1474c7221b6be95c984ab9759f85df8"}
Nov 03 17:41:06.356165 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:06.356022    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds" event={"ID":"c815c4f2-540a-4951-910a-b75c9ad9a72d","Type":"ContainerStarted","Data":"56273d304634eb3f6eac2372a2df1535d9080cbf43892d7796f6c7e3cae16dcb"}
Nov 03 17:41:06.553984 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:06.553863    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds" podStartSLOduration=2.5538422020000002 podStartE2EDuration="2.553842202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:06.553212107 +0000 UTC m=+402.563950202" watchObservedRunningTime="2026-11-03 17:41:06.553842202 +0000 UTC m=+402.564580276"
Nov 03 17:41:06.580419 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:06.580291    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:06.847762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:06.847649    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:07.030824 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:07.030756    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:07.199328 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:07.199283    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:07.206237 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:07.206207    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:07.362770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:07.362389    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:07.363409 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:07.363385    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85rq9ds"
Nov 03 17:41:08.387414 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:08.387371    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:08.542187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:08.542151    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:09.833361 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:09.832666    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:09.997762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:09.997700    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:10.339204 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:41:10.339171    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:41:10.381545 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:10.381511    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr" event={"ID":"ac802f81-9b7f-4abe-9b66-be17114d4a7c","Type":"ContainerStarted","Data":"9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03"}
Nov 03 17:41:10.381743 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:10.381575    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:10.605561 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:10.605441    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr" podStartSLOduration=2.721505584 podStartE2EDuration="7.605425143s" totalImagesPullingTime="4.883919559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:10.554023101 +0000 UTC m=+406.564761198" watchObservedRunningTime="2026-11-03 17:41:10.605425143 +0000 UTC m=+406.616163215"
Nov 03 17:41:11.228225 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.228184    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:41:11.231971 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.231949    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.288064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.288032    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 17:41:11.293980 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.293940    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:41:11.306493 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.306464    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0456b517-d0a7-4fee-b0f3-635cbad921e7-config-file\") pod \"limitador-limitador-7c9b87d574-84n8v\" (UID: \"0456b517-d0a7-4fee-b0f3-635cbad921e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.306493 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.306495    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgfk\" (UniqueName: \"kubernetes.io/projected/0456b517-d0a7-4fee-b0f3-635cbad921e7-kube-api-access-4tgfk\") pod \"limitador-limitador-7c9b87d574-84n8v\" (UID: \"0456b517-d0a7-4fee-b0f3-635cbad921e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.407723 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.407687    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0456b517-d0a7-4fee-b0f3-635cbad921e7-config-file\") pod \"limitador-limitador-7c9b87d574-84n8v\" (UID: \"0456b517-d0a7-4fee-b0f3-635cbad921e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.407723 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.407724    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgfk\" (UniqueName: \"kubernetes.io/projected/0456b517-d0a7-4fee-b0f3-635cbad921e7-kube-api-access-4tgfk\") pod \"limitador-limitador-7c9b87d574-84n8v\" (UID: \"0456b517-d0a7-4fee-b0f3-635cbad921e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.408373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.408341    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0456b517-d0a7-4fee-b0f3-635cbad921e7-config-file\") pod \"limitador-limitador-7c9b87d574-84n8v\" (UID: \"0456b517-d0a7-4fee-b0f3-635cbad921e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.473579 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.473539    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgfk\" (UniqueName: \"kubernetes.io/projected/0456b517-d0a7-4fee-b0f3-635cbad921e7-kube-api-access-4tgfk\") pod \"limitador-limitador-7c9b87d574-84n8v\" (UID: \"0456b517-d0a7-4fee-b0f3-635cbad921e7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.535208 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.535124    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:41:11.545433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.545405    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:11.647621 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.647576    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:11.925858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:11.925831    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:41:11.927714 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:41:11.927684    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0456b517_d0a7_4fee_b0f3_635cbad921e7.slice/crio-7367b8db2a426fcc47b80d78bd2b645fb6af2399cb155c82bd734bfbf66cbd8b WatchSource:0}: Error finding container 7367b8db2a426fcc47b80d78bd2b645fb6af2399cb155c82bd734bfbf66cbd8b: Status 404 returned error can't find the container with id 7367b8db2a426fcc47b80d78bd2b645fb6af2399cb155c82bd734bfbf66cbd8b
Nov 03 17:41:12.393784 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:12.393754    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v" event={"ID":"0456b517-d0a7-4fee-b0f3-635cbad921e7","Type":"ContainerStarted","Data":"7367b8db2a426fcc47b80d78bd2b645fb6af2399cb155c82bd734bfbf66cbd8b"}
Nov 03 17:41:12.436130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:12.436096    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:12.622379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:12.622342    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:14.406195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:14.406154    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v" event={"ID":"0456b517-d0a7-4fee-b0f3-635cbad921e7","Type":"ContainerStarted","Data":"dea60b25f1ebcced5219947c5c4c65d8a0fe277a2cc2c9bc01c869746d1aa180"}
Nov 03 17:41:14.406604 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:14.406259    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:14.533311 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:14.533244    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v" podStartSLOduration=1.3466204130000001 podStartE2EDuration="3.533223847s" totalImagesPullingTime="2.186603434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:14.53100173 +0000 UTC m=+410.541739819" watchObservedRunningTime="2026-11-03 17:41:14.533223847 +0000 UTC m=+410.543961920"
Nov 03 17:41:15.687013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:15.686967    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:15.879573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:15.879535    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:20.119408 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:20.119366    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:20.276299 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:20.276262    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:20.451844 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:20.451810    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:20.611893 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:20.611863    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:21.389387 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:21.389358    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 17:41:22.408463 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:22.408415    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:22.596536 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:22.596494    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:22.755387 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:22.755300    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:25.413271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:25.413240    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-84n8v"
Nov 03 17:41:29.227859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:29.227827    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:29.386298 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:29.386268    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:30.433154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:30.433114    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:30.596109 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:30.596072    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:32.709517 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:32.709483    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:40.338825 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:41:40.338783    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:41:50.331021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:50.330967    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:50.518503 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:50.518466    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:51.207058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:51.207017    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:54.312720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:54.312676    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:54.473041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:41:54.473003    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:10.338189 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:42:10.338107    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:42:13.966879 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:13.966843    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:14.135474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:14.135420    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:14.309233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:14.309150    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:14.478035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:14.478004    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:14.641048 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:14.641017    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:19.012111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:19.012075    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:19.179450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:19.179416    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:19.369131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:19.369036    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:19.577439 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:19.577406    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:23.931065 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:23.931026    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:24.172548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:24.172509    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:25.312029 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:25.311989    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:25.479627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:25.479596    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:25.647413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:25.647372    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:25.804581 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:25.804540    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:25.967129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:25.967050    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:26.245220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.245137    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:26.254271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.254242    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g6hps"]
Nov 03 17:42:26.256881 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.256865    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.314154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.314125    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 17:42:26.357787 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.357707    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48de8580-21aa-40d3-add8-03bcdfdeba4d-cert\") pod \"kserve-controller-manager-776cc66d47-g6hps\" (UID: \"48de8580-21aa-40d3-add8-03bcdfdeba4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.358037 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.357845    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwkp\" (UniqueName: \"kubernetes.io/projected/48de8580-21aa-40d3-add8-03bcdfdeba4d-kube-api-access-hvwkp\") pod \"kserve-controller-manager-776cc66d47-g6hps\" (UID: \"48de8580-21aa-40d3-add8-03bcdfdeba4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.361396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.361361    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 17:42:26.361396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.361372    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 17:42:26.361589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.361460    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bbrrr\""
Nov 03 17:42:26.363676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.363656    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g6hps"]
Nov 03 17:42:26.416343 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.416306    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:26.458760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.458688    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48de8580-21aa-40d3-add8-03bcdfdeba4d-cert\") pod \"kserve-controller-manager-776cc66d47-g6hps\" (UID: \"48de8580-21aa-40d3-add8-03bcdfdeba4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.458962 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.458799    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwkp\" (UniqueName: \"kubernetes.io/projected/48de8580-21aa-40d3-add8-03bcdfdeba4d-kube-api-access-hvwkp\") pod \"kserve-controller-manager-776cc66d47-g6hps\" (UID: \"48de8580-21aa-40d3-add8-03bcdfdeba4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.461505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.461478    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48de8580-21aa-40d3-add8-03bcdfdeba4d-cert\") pod \"kserve-controller-manager-776cc66d47-g6hps\" (UID: \"48de8580-21aa-40d3-add8-03bcdfdeba4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.522480 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.522399    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwkp\" (UniqueName: \"kubernetes.io/projected/48de8580-21aa-40d3-add8-03bcdfdeba4d-kube-api-access-hvwkp\") pod \"kserve-controller-manager-776cc66d47-g6hps\" (UID: \"48de8580-21aa-40d3-add8-03bcdfdeba4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.568758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.568692    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:26.579150 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.579116    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:26.946355 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:26.946331    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g6hps"]
Nov 03 17:42:26.948213 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:42:26.948179    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48de8580_21aa_40d3_add8_03bcdfdeba4d.slice/crio-c666491be3bb44dc6044480cd6c9ca9d87daf446d8991844557cd26a60b1beef WatchSource:0}: Error finding container c666491be3bb44dc6044480cd6c9ca9d87daf446d8991844557cd26a60b1beef: Status 404 returned error can't find the container with id c666491be3bb44dc6044480cd6c9ca9d87daf446d8991844557cd26a60b1beef
Nov 03 17:42:27.657613 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:27.657578    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:27.817822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:27.817784    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps" event={"ID":"48de8580-21aa-40d3-add8-03bcdfdeba4d","Type":"ContainerStarted","Data":"c666491be3bb44dc6044480cd6c9ca9d87daf446d8991844557cd26a60b1beef"}
Nov 03 17:42:30.837481 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:30.837439    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps" event={"ID":"48de8580-21aa-40d3-add8-03bcdfdeba4d","Type":"ContainerStarted","Data":"29099c5adbfc4cf56e126b2cf52e98f4be8d95cbe0c15148f87a6a87f095237e"}
Nov 03 17:42:30.837918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:30.837542    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:42:30.958002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:30.957952    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps" podStartSLOduration=1.9339042210000001 podStartE2EDuration="4.957928281s" totalImagesPullingTime="3.02402406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:30.95710064 +0000 UTC m=+486.967838734" watchObservedRunningTime="2026-11-03 17:42:30.957928281 +0000 UTC m=+486.968666352"
Nov 03 17:42:33.004922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:33.004886    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:33.158631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:33.158599    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:33.347569 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:33.347484    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:33.517669 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:33.517634    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:33.697367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:33.697334    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:33.853423 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:33.853393    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:34.037417 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:34.037331    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:34.302935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:34.302849    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:34.512466 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:34.512425    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:40.338432 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:42:40.338395    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:42:52.711611 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:52.711555    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:52.947522 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:52.947475    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:54.786332 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:54.786292    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:55.002256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:55.002209    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:55.173981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:55.173933    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:55.335769 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:42:55.335707    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:01.821930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:01.821888    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:43:01.847229 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:01.847198    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-g6hps"
Nov 03 17:43:02.137709 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:02.137610    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:06.222038 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.221999    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:06.422158 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.422125    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:06.647107 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.647072    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"]
Nov 03 17:43:06.650622 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.650605    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.660784 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.660759    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.660927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.660882    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.660995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.660938    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.660995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.660964    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvlx\" (UniqueName: \"kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.711152 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.711118    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:43:06.712280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.712222    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:43:06.761489 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761452    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 17:43:06.761489 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761478    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fhmsl\""
Nov 03 17:43:06.761725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761513    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 17:43:06.761812 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761787    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.761868 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761856    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.761915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761876    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvlx\" (UniqueName: \"kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.761965 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.761944    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.762506 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.762484    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.763943 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.763921    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"]
Nov 03 17:43:06.764485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.764465    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.764485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.764478    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.828530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.828496    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvlx\" (UniqueName: \"kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx\") pod \"maas-api-6cc56655c-jkbj7\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:06.963722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:06.963628    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:07.367266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:07.367243    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"]
Nov 03 17:43:07.369509 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:43:07.369478    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878c563c_ed19_4910_8aef_023edd6e9376.slice/crio-b5ff72575c8450c0a1f7b9164a3a02eeb838739481a436b4eee8d6a8500e1570 WatchSource:0}: Error finding container b5ff72575c8450c0a1f7b9164a3a02eeb838739481a436b4eee8d6a8500e1570: Status 404 returned error can't find the container with id b5ff72575c8450c0a1f7b9164a3a02eeb838739481a436b4eee8d6a8500e1570
Nov 03 17:43:08.063253 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:08.063216    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" event={"ID":"878c563c-ed19-4910-8aef-023edd6e9376","Type":"ContainerStarted","Data":"b5ff72575c8450c0a1f7b9164a3a02eeb838739481a436b4eee8d6a8500e1570"}
Nov 03 17:43:10.339344 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:43:10.339301    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:43:11.084465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.084355    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" event={"ID":"878c563c-ed19-4910-8aef-023edd6e9376","Type":"ContainerStarted","Data":"44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d"}
Nov 03 17:43:11.084465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.084438    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:11.086080 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.086054    5838 generic.go:356] "Generic (PLEG): container finished" podID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerID="cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411" exitCode=0
Nov 03 17:43:11.086218 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.086092    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" event={"ID":"3f4f6054-05f5-41c5-848f-b245cbff25d3","Type":"ContainerDied","Data":"cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"}
Nov 03 17:43:11.086444 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.086424    5838 scope.go:122] "RemoveContainer" containerID="cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"
Nov 03 17:43:11.087538 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.087510    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:43:11.087596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.087583    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:43:11.087633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.087615    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:43:11.207164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:11.207105    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" podStartSLOduration=2.221730772 podStartE2EDuration="5.207086856s" totalImagesPullingTime="2.985356084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:11.205934481 +0000 UTC m=+527.216672553" watchObservedRunningTime="2026-11-03 17:43:11.207086856 +0000 UTC m=+527.217824934"
Nov 03 17:43:12.093886 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:12.093837    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" event={"ID":"3f4f6054-05f5-41c5-848f-b245cbff25d3","Type":"ContainerStarted","Data":"601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b"}
Nov 03 17:43:12.633238 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:12.633197    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:43:12.634450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:12.634421    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jv82l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:43:12.634577 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:12.634476    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 17:43:12.819759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:12.812858    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:43:13.633116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:13.633077    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jv82l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:43:13.633528 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:13.633173    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 17:43:14.632570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:14.632530    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jv82l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:43:14.632773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:14.632591    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 17:43:15.633130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:15.633091    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jv82l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:43:15.633550 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:15.633155    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 17:43:16.633271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:16.633226    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jv82l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:43:16.633686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:16.633287    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 17:43:17.099447 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:17.099415    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 17:43:17.637294 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:17.637262    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:43:17.638572 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:17.638546    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:43:33.439697 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:33.439620    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:38.056919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:43:38.056877    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:40.337886 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:43:40.337848    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:44:10.338897 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:44:10.338856    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:44:14.898882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:14.898841    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:15.070938 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:15.070904    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:24.217573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:24.217541    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:44:24.223088 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:24.223065    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:44:25.073480 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:25.073442    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:25.267324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:25.267292    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:33.821259 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:33.821226    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:40.338469 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:44:40.338430    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:44:45.842421 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:45.842383    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:49.575634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:49.575593    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:44:49.874177 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:49.874095    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:53.990353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:53.990319    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:58.171016 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:58.170984    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:58.346030 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:44:58.345996    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:45:05.429827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.429747    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"]
Nov 03 17:45:05.436586 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.436563    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.495542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.495511    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"]
Nov 03 17:45:05.502153 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.502124    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fzx\" (UniqueName: \"kubernetes.io/projected/00ed915e-64f7-4efd-981b-71d7c8756b48-kube-api-access-72fzx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.502327 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.502162    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.502327 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.502233    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.502327 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.502261    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.502465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.502371    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.502465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.502434    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00ed915e-64f7-4efd-981b-71d7c8756b48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.533222 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.533172    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:05.543101 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.543071    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 17:45:05.543241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.543145    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 17:45:05.543241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.543155    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-52z8v\""
Nov 03 17:45:05.543241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.543149    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 17:45:05.603697 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.603652    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.603919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.603710    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.603919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.603804    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00ed915e-64f7-4efd-981b-71d7c8756b48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.603919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.603876    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fzx\" (UniqueName: \"kubernetes.io/projected/00ed915e-64f7-4efd-981b-71d7c8756b48-kube-api-access-72fzx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.603919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.603902    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.604127 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.603947    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.604355 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.604329    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.604499 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.604327    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.604745 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.604514    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.606280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.606254    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00ed915e-64f7-4efd-981b-71d7c8756b48-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.606685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.606662    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00ed915e-64f7-4efd-981b-71d7c8756b48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.671273 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.671245    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fzx\" (UniqueName: \"kubernetes.io/projected/00ed915e-64f7-4efd-981b-71d7c8756b48-kube-api-access-72fzx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj\" (UID: \"00ed915e-64f7-4efd-981b-71d7c8756b48\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:05.754595 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:05.754504    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:06.180301 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:06.180278    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"]
Nov 03 17:45:06.182253 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:45:06.182217    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ed915e_64f7_4efd_981b_71d7c8756b48.slice/crio-9af0beb30567048c49151d4a919fcd711806f581f63ee5c4bd2ec5b30847a2c8 WatchSource:0}: Error finding container 9af0beb30567048c49151d4a919fcd711806f581f63ee5c4bd2ec5b30847a2c8: Status 404 returned error can't find the container with id 9af0beb30567048c49151d4a919fcd711806f581f63ee5c4bd2ec5b30847a2c8
Nov 03 17:45:06.745530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:06.745496    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj" event={"ID":"00ed915e-64f7-4efd-981b-71d7c8756b48","Type":"ContainerStarted","Data":"9af0beb30567048c49151d4a919fcd711806f581f63ee5c4bd2ec5b30847a2c8"}
Nov 03 17:45:08.759531 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:08.759491    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj" event={"ID":"00ed915e-64f7-4efd-981b-71d7c8756b48","Type":"ContainerStarted","Data":"cf112be05191bde8788249c7226810154e3543c6bd947927192938f65b9e58ec"}
Nov 03 17:45:08.892838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:08.892781    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj" podStartSLOduration=1.836883472 podStartE2EDuration="3.892763587s" totalImagesPullingTime="2.055880115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:08.892270707 +0000 UTC m=+644.903008781" watchObservedRunningTime="2026-11-03 17:45:08.892763587 +0000 UTC m=+644.903501658"
Nov 03 17:45:09.322386 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.322352    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:45:09.372691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.372648    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"]
Nov 03 17:45:09.376409 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.376384    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.431543 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.431511    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 17:45:09.453064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.453031    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"]
Nov 03 17:45:09.474340 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.474304    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:09.542650 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.542616    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.542859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.542674    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.542859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.542703    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f956514d-9307-413e-a516-7d5449040eb8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.542935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.542888    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.542935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.542930    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.543002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.542964    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqxt\" (UniqueName: \"kubernetes.io/projected/f956514d-9307-413e-a516-7d5449040eb8-kube-api-access-jgqxt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.643878 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.643849    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.643893    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqxt\" (UniqueName: \"kubernetes.io/projected/f956514d-9307-413e-a516-7d5449040eb8-kube-api-access-jgqxt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.643934    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.643978    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.643998    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f956514d-9307-413e-a516-7d5449040eb8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.644033    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.644341    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644429 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.644399    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.644490 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.644439    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.646482 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.646455    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f956514d-9307-413e-a516-7d5449040eb8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.646892 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.646870    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f956514d-9307-413e-a516-7d5449040eb8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.707718 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.707679    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqxt\" (UniqueName: \"kubernetes.io/projected/f956514d-9307-413e-a516-7d5449040eb8-kube-api-access-jgqxt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qrswt\" (UID: \"f956514d-9307-413e-a516-7d5449040eb8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:09.754684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.754647    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:09.990359 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:09.990251    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:10.340886 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:45:10.340795    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:45:10.394571 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:10.394548    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"]
Nov 03 17:45:10.396905 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:45:10.396869    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf956514d_9307_413e_a516_7d5449040eb8.slice/crio-b0aa2bf966c6f022b99550856bdbbc50c72983892e89c7212b2359776b13da8e WatchSource:0}: Error finding container b0aa2bf966c6f022b99550856bdbbc50c72983892e89c7212b2359776b13da8e: Status 404 returned error can't find the container with id b0aa2bf966c6f022b99550856bdbbc50c72983892e89c7212b2359776b13da8e
Nov 03 17:45:10.772715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:10.772665    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt" event={"ID":"f956514d-9307-413e-a516-7d5449040eb8","Type":"ContainerStarted","Data":"b0aa2bf966c6f022b99550856bdbbc50c72983892e89c7212b2359776b13da8e"}
Nov 03 17:45:11.780432 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:11.780396    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt" event={"ID":"f956514d-9307-413e-a516-7d5449040eb8","Type":"ContainerStarted","Data":"f1cdc5c4b333e60e7dc4834149780d67119df227023a36e23f2d06fc75c5d228"}
Nov 03 17:45:11.903487 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:11.903401    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt" podStartSLOduration=2.291483324 podStartE2EDuration="2.903385917s" totalImagesPullingTime="611.902593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:11.901386081 +0000 UTC m=+647.912124198" watchObservedRunningTime="2026-11-03 17:45:11.903385917 +0000 UTC m=+647.914124025"
Nov 03 17:45:11.990599 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:11.990563    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:13.440631 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.440597    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"]
Nov 03 17:45:13.444339 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.444317    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.546437 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.546393    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:13.548239 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.548219    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 17:45:13.550570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.550545    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"]
Nov 03 17:45:13.586666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.586627    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.586857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.586691    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.586857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.586788    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.586857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.586835    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r4ql\" (UniqueName: \"kubernetes.io/projected/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-kube-api-access-4r4ql\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.586966 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.586914    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.587000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.586962    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688452 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688410    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688470    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688489    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688525    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r4ql\" (UniqueName: \"kubernetes.io/projected/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-kube-api-access-4r4ql\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688551    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688571    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688929    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.688981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.688933    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.689048 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.689004    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.691135 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.691071    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.691318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.691296    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.752700 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.752661    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r4ql\" (UniqueName: \"kubernetes.io/projected/6ebaa47f-2620-4fb8-8843-eb8419cf4c2a-kube-api-access-4r4ql\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-plqxs\" (UID: \"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:13.757655 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:13.757634    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:14.145873 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:14.145850    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"]
Nov 03 17:45:14.147709 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:45:14.147680    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebaa47f_2620_4fb8_8843_eb8419cf4c2a.slice/crio-e9c233db45357c3b206d469126856125dc834deae54cadb7c5ab1be1d749e648 WatchSource:0}: Error finding container e9c233db45357c3b206d469126856125dc834deae54cadb7c5ab1be1d749e648: Status 404 returned error can't find the container with id e9c233db45357c3b206d469126856125dc834deae54cadb7c5ab1be1d749e648
Nov 03 17:45:14.802442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:14.802410    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs" event={"ID":"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a","Type":"ContainerStarted","Data":"e9c233db45357c3b206d469126856125dc834deae54cadb7c5ab1be1d749e648"}
Nov 03 17:45:15.755660 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:15.755622    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:15.774278 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:15.774250    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:15.809239 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:15.809202    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs" event={"ID":"6ebaa47f-2620-4fb8-8843-eb8419cf4c2a","Type":"ContainerStarted","Data":"11cbbfd040e7f7542ef4dee5c9f1b37b803d9b7431c6e10296a7eee9d5ea737a"}
Nov 03 17:45:15.820581 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:15.820548    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj"
Nov 03 17:45:16.153372 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:16.153318    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs" podStartSLOduration=2.545333025 podStartE2EDuration="3.153302646s" totalImagesPullingTime="607.969621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:16.152691978 +0000 UTC m=+652.163430051" watchObservedRunningTime="2026-11-03 17:45:16.153302646 +0000 UTC m=+652.164040718"
Nov 03 17:45:16.758832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:16.758800    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:17.520147 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:17.520112    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:19.991008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:19.990966    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:20.003620 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:20.003593    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:20.853815 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:20.853788    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt"
Nov 03 17:45:23.758357 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:23.758311    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:23.770680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:23.770649    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:23.871399 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:23.871371    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-plqxs"
Nov 03 17:45:25.854068 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:25.854033    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:45:33.692402 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:33.692359    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:34.113494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:34.113405    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:40.338144 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:45:40.338107    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:45:42.016330 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.016297    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:42.020129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.020104    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6695677c69-pbd77"]
Nov 03 17:45:42.023896 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.023880    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.127845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.127806    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 17:45:42.128050 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.127817    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nxqd7\""
Nov 03 17:45:42.128050 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.127853    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 17:45:42.130600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.130573    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6695677c69-pbd77"]
Nov 03 17:45:42.176700 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.176669    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7hk\" (UniqueName: \"kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk\") pod \"ai-gateway-controller-6695677c69-pbd77\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") " pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.176905 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.176721    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs\") pod \"ai-gateway-controller-6695677c69-pbd77\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") " pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.181057 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.181023    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:42.277338 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.277242    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs\") pod \"ai-gateway-controller-6695677c69-pbd77\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") " pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.277505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.277360    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7hk\" (UniqueName: \"kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk\") pod \"ai-gateway-controller-6695677c69-pbd77\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") " pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.279957 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.279938    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs\") pod \"ai-gateway-controller-6695677c69-pbd77\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") " pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.341339 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.341310    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7hk\" (UniqueName: \"kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk\") pod \"ai-gateway-controller-6695677c69-pbd77\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") " pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:42.636335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:42.636253    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:43.018159 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:43.018132    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6695677c69-pbd77"]
Nov 03 17:45:43.020565 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:45:43.020532    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73c5ed3_1860_43fb_a742_9df609c3f030.slice/crio-f873a7cab1bd493fa5996aca1a3c98c74184d680cd0b81d3a588971cb89b72eb WatchSource:0}: Error finding container f873a7cab1bd493fa5996aca1a3c98c74184d680cd0b81d3a588971cb89b72eb: Status 404 returned error can't find the container with id f873a7cab1bd493fa5996aca1a3c98c74184d680cd0b81d3a588971cb89b72eb
Nov 03 17:45:43.022575 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:43.022557    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:45:43.991428 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:43.991386    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77" event={"ID":"c73c5ed3-1860-43fb-a742-9df609c3f030","Type":"ContainerStarted","Data":"f873a7cab1bd493fa5996aca1a3c98c74184d680cd0b81d3a588971cb89b72eb"}
Nov 03 17:45:48.506005 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.505961    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"]
Nov 03 17:45:48.510426 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.510401    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.606980 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.606928    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"]
Nov 03 17:45:48.640426 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640392    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640426 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640427    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640641 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640449    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640641 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640588    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640641 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640619    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rc9c\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640748 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640641    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640801 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640769    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640808    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.640922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.640901    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742391 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742343    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742406    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742430    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742459    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742509    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742536    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rc9c\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742717    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742806    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742863    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.742939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742905    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.743159 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.742952    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.743407 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.743352    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.743407 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.743391    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.744762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.744711    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.746269 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.746246    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.746898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.746873    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.809501 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.809417    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.809662 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.809583    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rc9c\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c\") pod \"maas-default-gateway-openshift-default-5df48fcc99-5tvsl\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:48.825270 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:48.825232    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:49.262965 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:49.262942    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"]
Nov 03 17:45:49.264567 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:45:49.264542    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed72a79_ce64_41f0_b186_56b6a94821ef.slice/crio-79a0e4f3b642fee8e4353c317963979acc89b553c5342ab68bae1f2937e44694 WatchSource:0}: Error finding container 79a0e4f3b642fee8e4353c317963979acc89b553c5342ab68bae1f2937e44694: Status 404 returned error can't find the container with id 79a0e4f3b642fee8e4353c317963979acc89b553c5342ab68bae1f2937e44694
Nov 03 17:45:49.294801 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:49.294724    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:49.294955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:49.294857    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:49.294955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:49.294901    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:50.032080 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.032039    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" event={"ID":"4ed72a79-ce64-41f0-b186-56b6a94821ef","Type":"ContainerStarted","Data":"d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8"}
Nov 03 17:45:50.032548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.032103    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" event={"ID":"4ed72a79-ce64-41f0-b186-56b6a94821ef","Type":"ContainerStarted","Data":"79a0e4f3b642fee8e4353c317963979acc89b553c5342ab68bae1f2937e44694"}
Nov 03 17:45:50.033668 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.033646    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77" event={"ID":"c73c5ed3-1860-43fb-a742-9df609c3f030","Type":"ContainerStarted","Data":"8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902"}
Nov 03 17:45:50.033768 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.033714    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:45:50.161679 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.161628    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podStartSLOduration=2.161613678 podStartE2EDuration="2.161613678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:50.160796442 +0000 UTC m=+686.171534538" watchObservedRunningTime="2026-11-03 17:45:50.161613678 +0000 UTC m=+686.172351747"
Nov 03 17:45:50.288841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.288715    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77" podStartSLOduration=3.00748229 podStartE2EDuration="9.288701236s" totalImagesPullingTime="6.281218946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:50.287173125 +0000 UTC m=+686.297911217" watchObservedRunningTime="2026-11-03 17:45:50.288701236 +0000 UTC m=+686.299439312"
Nov 03 17:45:50.826981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.826944    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:50.828322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.828297    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:50.828466 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:50.828347    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:51.826332 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:51.826294    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:51.826802 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:51.826361    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:52.826365 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:52.826323    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:52.826903 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:52.826393    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:53.428681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.428643    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:53.595179 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.595144    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:53.770870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.770774    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"]
Nov 03 17:45:53.774705 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.774685    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.785686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.785654    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ffdf\" (UniqueName: \"kubernetes.io/projected/1b848dc6-7e5e-46c2-b1e6-63fa37e718ac-kube-api-access-4ffdf\") pod \"payload-pre-processing-7b98bb678c-gmtqs\" (UID: \"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.785862 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.785709    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b848dc6-7e5e-46c2-b1e6-63fa37e718ac-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gmtqs\" (UID: \"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.825832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.825791    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:53.826035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.825856    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:53.834442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.834399    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"]
Nov 03 17:45:53.876647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.876618    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 17:45:53.886337 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.886308    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b848dc6-7e5e-46c2-b1e6-63fa37e718ac-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gmtqs\" (UID: \"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.886438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.886397    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ffdf\" (UniqueName: \"kubernetes.io/projected/1b848dc6-7e5e-46c2-b1e6-63fa37e718ac-kube-api-access-4ffdf\") pod \"payload-pre-processing-7b98bb678c-gmtqs\" (UID: \"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.887112 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.887093    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b848dc6-7e5e-46c2-b1e6-63fa37e718ac-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gmtqs\" (UID: \"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.937828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.937792    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-46szt"]
Nov 03 17:45:53.941718 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.941698    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:53.952108 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.952080    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ffdf\" (UniqueName: \"kubernetes.io/projected/1b848dc6-7e5e-46c2-b1e6-63fa37e718ac-kube-api-access-4ffdf\") pod \"payload-pre-processing-7b98bb678c-gmtqs\" (UID: \"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:53.986971 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.986927    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgmr\" (UniqueName: \"kubernetes.io/projected/25ceda13-c002-4ee6-9926-68b0e3a98cb9-kube-api-access-vjgmr\") pod \"payload-processing-55fbbbf9dc-46szt\" (UID: \"25ceda13-c002-4ee6-9926-68b0e3a98cb9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:53.987154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.987072    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceda13-c002-4ee6-9926-68b0e3a98cb9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-46szt\" (UID: \"25ceda13-c002-4ee6-9926-68b0e3a98cb9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:53.987154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:53.987088    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-46szt"]
Nov 03 17:45:54.004072 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.004040    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wtz5k\""
Nov 03 17:45:54.087584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.087490    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgmr\" (UniqueName: \"kubernetes.io/projected/25ceda13-c002-4ee6-9926-68b0e3a98cb9-kube-api-access-vjgmr\") pod \"payload-processing-55fbbbf9dc-46szt\" (UID: \"25ceda13-c002-4ee6-9926-68b0e3a98cb9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:54.087795 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.087637    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceda13-c002-4ee6-9926-68b0e3a98cb9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-46szt\" (UID: \"25ceda13-c002-4ee6-9926-68b0e3a98cb9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:54.088326 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.088300    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceda13-c002-4ee6-9926-68b0e3a98cb9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-46szt\" (UID: \"25ceda13-c002-4ee6-9926-68b0e3a98cb9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:54.089217 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.089194    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:54.151507 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.151480    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgmr\" (UniqueName: \"kubernetes.io/projected/25ceda13-c002-4ee6-9926-68b0e3a98cb9-kube-api-access-vjgmr\") pod \"payload-processing-55fbbbf9dc-46szt\" (UID: \"25ceda13-c002-4ee6-9926-68b0e3a98cb9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:54.256114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.256079    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:54.495565 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.495533    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"]
Nov 03 17:45:54.657510 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.657481    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-46szt"]
Nov 03 17:45:54.659390 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:45:54.659361    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ceda13_c002_4ee6_9926_68b0e3a98cb9.slice/crio-16bc2e3958d1563d58a57d8883462affeae9718b767685f588e0732262bfa4da WatchSource:0}: Error finding container 16bc2e3958d1563d58a57d8883462affeae9718b767685f588e0732262bfa4da: Status 404 returned error can't find the container with id 16bc2e3958d1563d58a57d8883462affeae9718b767685f588e0732262bfa4da
Nov 03 17:45:54.826220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.826138    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:54.826220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:54.826192    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:55.062295 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:55.062258    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt" event={"ID":"25ceda13-c002-4ee6-9926-68b0e3a98cb9","Type":"ContainerStarted","Data":"16bc2e3958d1563d58a57d8883462affeae9718b767685f588e0732262bfa4da"}
Nov 03 17:45:55.063666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:55.063639    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs" event={"ID":"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac","Type":"ContainerStarted","Data":"c1d5edec8361610f704e1160e8915e549c4e149322b9c89bc12080ec89646f10"}
Nov 03 17:45:55.827136 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:55.827096    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:55.827349 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:55.827151    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:56.826022 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:56.825986    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:45:56.826477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:56.826046    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 17:45:58.089277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.089239    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt" event={"ID":"25ceda13-c002-4ee6-9926-68b0e3a98cb9","Type":"ContainerStarted","Data":"50bb5dea685e9bd12b5f797687ea242139e027022cfd044cb2fc061928dc56c3"}
Nov 03 17:45:58.089885 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.089359    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:45:58.091157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.091127    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs" event={"ID":"1b848dc6-7e5e-46c2-b1e6-63fa37e718ac","Type":"ContainerStarted","Data":"5fc8426a133b9b1bbaac0e8e8366dec79c384a5b8f1fe5cd1af9caadae5824da"}
Nov 03 17:45:58.091255 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.091222    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:45:58.209378 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.209309    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt" podStartSLOduration=2.173590977 podStartE2EDuration="5.209289112s" totalImagesPullingTime="3.035698135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:58.208864848 +0000 UTC m=+694.219602946" watchObservedRunningTime="2026-11-03 17:45:58.209289112 +0000 UTC m=+694.220027185"
Nov 03 17:45:58.284851 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.284821    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:58.285149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.285128    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:58.286001 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.285980    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:45:58.329842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:45:58.329788    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs" podStartSLOduration=2.213507645 podStartE2EDuration="5.329770966s" totalImagesPullingTime="3.116263321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:58.328443886 +0000 UTC m=+694.339181978" watchObservedRunningTime="2026-11-03 17:45:58.329770966 +0000 UTC m=+694.340509041"
Nov 03 17:46:01.041126 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.041098    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:46:01.434470 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.434431    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-788f8bd568-8vqvp"]
Nov 03 17:46:01.438426 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.438408    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.477019 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.476983    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87d86\" (UniqueName: \"kubernetes.io/projected/edeb0b48-390f-43f0-9e5b-e14a7ff76626-kube-api-access-87d86\") pod \"ai-gateway-controller-788f8bd568-8vqvp\" (UID: \"edeb0b48-390f-43f0-9e5b-e14a7ff76626\") " pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.477198 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.477115    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeb0b48-390f-43f0-9e5b-e14a7ff76626-metrics-certs\") pod \"ai-gateway-controller-788f8bd568-8vqvp\" (UID: \"edeb0b48-390f-43f0-9e5b-e14a7ff76626\") " pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.498887 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.498856    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-788f8bd568-8vqvp"]
Nov 03 17:46:01.578350 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.578316    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeb0b48-390f-43f0-9e5b-e14a7ff76626-metrics-certs\") pod \"ai-gateway-controller-788f8bd568-8vqvp\" (UID: \"edeb0b48-390f-43f0-9e5b-e14a7ff76626\") " pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.578535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.578423    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d86\" (UniqueName: \"kubernetes.io/projected/edeb0b48-390f-43f0-9e5b-e14a7ff76626-kube-api-access-87d86\") pod \"ai-gateway-controller-788f8bd568-8vqvp\" (UID: \"edeb0b48-390f-43f0-9e5b-e14a7ff76626\") " pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.581071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.581043    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeb0b48-390f-43f0-9e5b-e14a7ff76626-metrics-certs\") pod \"ai-gateway-controller-788f8bd568-8vqvp\" (UID: \"edeb0b48-390f-43f0-9e5b-e14a7ff76626\") " pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.644661 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.644628    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d86\" (UniqueName: \"kubernetes.io/projected/edeb0b48-390f-43f0-9e5b-e14a7ff76626-kube-api-access-87d86\") pod \"ai-gateway-controller-788f8bd568-8vqvp\" (UID: \"edeb0b48-390f-43f0-9e5b-e14a7ff76626\") " pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:01.752848 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:01.752754    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:02.131724 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:02.131699    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-788f8bd568-8vqvp"]
Nov 03 17:46:02.133492 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:46:02.133458    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeb0b48_390f_43f0_9e5b_e14a7ff76626.slice/crio-4b09eec715905c8e5d27b23c7f92e918fbcf75b47432f6b60d85d31f2d81eeb2 WatchSource:0}: Error finding container 4b09eec715905c8e5d27b23c7f92e918fbcf75b47432f6b60d85d31f2d81eeb2: Status 404 returned error can't find the container with id 4b09eec715905c8e5d27b23c7f92e918fbcf75b47432f6b60d85d31f2d81eeb2
Nov 03 17:46:03.124840 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:03.124801    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp" event={"ID":"edeb0b48-390f-43f0-9e5b-e14a7ff76626","Type":"ContainerStarted","Data":"587ed44c956e75c043f32bf1fbe94a9aea6eeb80ae3a9e144b00a0918378903b"}
Nov 03 17:46:03.125042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:03.124881    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp" event={"ID":"edeb0b48-390f-43f0-9e5b-e14a7ff76626","Type":"ContainerStarted","Data":"4b09eec715905c8e5d27b23c7f92e918fbcf75b47432f6b60d85d31f2d81eeb2"}
Nov 03 17:46:03.125042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:03.125013    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:03.247304 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:03.247241    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp" podStartSLOduration=1.370953884 podStartE2EDuration="2.247227128s" totalImagesPullingTime="876.273244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:03.245021655 +0000 UTC m=+699.255759728" watchObservedRunningTime="2026-11-03 17:46:03.247227128 +0000 UTC m=+699.257965201"
Nov 03 17:46:05.571654 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:05.571592    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:09.100584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:09.100555    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-46szt"
Nov 03 17:46:09.101074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:09.100623    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gmtqs"
Nov 03 17:46:09.817950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:09.817910    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:09.972727 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:09.972686    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:10.338863 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:46:10.338826    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:46:14.132185 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.132156    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-788f8bd568-8vqvp"
Nov 03 17:46:14.305327 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.305293    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6695677c69-pbd77"]
Nov 03 17:46:14.305551 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.305519    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77" podUID="c73c5ed3-1860-43fb-a742-9df609c3f030" containerName="manager" containerID="cri-o://8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902" gracePeriod=30
Nov 03 17:46:14.574182 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.574142    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:46:14.711241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.711148    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk\" (UniqueName: \"kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk\") pod \"c73c5ed3-1860-43fb-a742-9df609c3f030\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") "
Nov 03 17:46:14.711241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.711214    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs\") pod \"c73c5ed3-1860-43fb-a742-9df609c3f030\" (UID: \"c73c5ed3-1860-43fb-a742-9df609c3f030\") "
Nov 03 17:46:14.713781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.713711    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs" pod "c73c5ed3-1860-43fb-a742-9df609c3f030" (UID: "c73c5ed3-1860-43fb-a742-9df609c3f030"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:14.713921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.713822    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk" pod "c73c5ed3-1860-43fb-a742-9df609c3f030" (UID: "c73c5ed3-1860-43fb-a742-9df609c3f030"). InnerVolumeSpecName "kube-api-access-nd7hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:14.812282 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.812253    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7hk\" (UniqueName: \"kubernetes.io/projected/c73c5ed3-1860-43fb-a742-9df609c3f030-kube-api-access-nd7hk\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:46:14.812282 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:14.812280    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73c5ed3-1860-43fb-a742-9df609c3f030-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:46:15.194781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.194722    5838 generic.go:356] "Generic (PLEG): container finished" podID="c73c5ed3-1860-43fb-a742-9df609c3f030" containerID="8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902" exitCode=0
Nov 03 17:46:15.194781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.194767    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77" event={"ID":"c73c5ed3-1860-43fb-a742-9df609c3f030","Type":"ContainerDied","Data":"8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902"}
Nov 03 17:46:15.195294 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.194802    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77"
Nov 03 17:46:15.195294 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.194833    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6695677c69-pbd77" event={"ID":"c73c5ed3-1860-43fb-a742-9df609c3f030","Type":"ContainerDied","Data":"f873a7cab1bd493fa5996aca1a3c98c74184d680cd0b81d3a588971cb89b72eb"}
Nov 03 17:46:15.195294 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.194864    5838 scope.go:122] "RemoveContainer" containerID="8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902"
Nov 03 17:46:15.204717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.204699    5838 scope.go:122] "RemoveContainer" containerID="8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902"
Nov 03 17:46:15.205061 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:46:15.205033    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902\": container with ID starting with 8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902 not found: ID does not exist" containerID="8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902"
Nov 03 17:46:15.205118 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.205066    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902"} err="failed to get container status \"8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902\": rpc error: code = NotFound desc = could not find container \"8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902\": container with ID starting with 8f90c6e418715e891740f539f7e6afe0c971af7309d5ed01b17937ab1f57f902 not found: ID does not exist"
Nov 03 17:46:15.365069 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.365031    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6695677c69-pbd77"]
Nov 03 17:46:15.415276 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:15.415247    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6695677c69-pbd77"]
Nov 03 17:46:16.116161 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:16.116123    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73c5ed3-1860-43fb-a742-9df609c3f030" path="/var/lib/kubelet/pods/c73c5ed3-1860-43fb-a742-9df609c3f030/volumes"
Nov 03 17:46:16.279445 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:16.279409    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:16.912345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:16.912310    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:18.848845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:18.848800    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:19.131480 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:19.131379    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:20.814679 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:20.814647    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:20.986028 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:20.985994    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:34.459391 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:34.459298    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:40.338558 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:46:40.338516    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:46:47.414110 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.414052    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:47.580589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.580549    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:47.987401 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.987363    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"]
Nov 03 17:46:47.987927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.987912    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c73c5ed3-1860-43fb-a742-9df609c3f030" containerName="manager"
Nov 03 17:46:47.987977 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.987929    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73c5ed3-1860-43fb-a742-9df609c3f030" containerName="manager"
Nov 03 17:46:47.988037 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.988027    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73c5ed3-1860-43fb-a742-9df609c3f030" containerName="manager"
Nov 03 17:46:47.991411 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:47.991386    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.047439 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.047412    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-07f297-openshift-default-dockercfg-dfdjr\""
Nov 03 17:46:48.081362 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.081329    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.099233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.099199    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"]
Nov 03 17:46:48.130232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130198    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130411 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130296    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130411 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130359    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130411 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130389    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130576 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130417    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nml\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130576 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130493    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130605    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130673    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.130840 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.130722    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.231595 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231556    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.231835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231634    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.231835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231659    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.231835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231686    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.231835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231708    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.231835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231759    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nml\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231920    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.231966    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.232018    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.232050    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.232598    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232954 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.232631    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232954 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.232662    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.232954 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.232871    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.234162 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.234140    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.234662 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.234636    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.299201 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.299113    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.299361 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.299224    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nml\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.300277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.300250    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.305236 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.305213    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:48.459699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.459659    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.626336 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.626248    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.644594 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.644563    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"]
Nov 03 17:46:48.648562 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.648544    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.719167 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.719114    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-776ba7-openshift-default-dockercfg-nv8qc\""
Nov 03 17:46:48.723655 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.723620    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"]
Nov 03 17:46:48.737550 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.737515    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.737720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.737566    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xt9n\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.737720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.737685    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.737870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.737835    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.737930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.737913    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.738007 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.737986    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.738050 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.738031    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.738097 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.738086    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.738153 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.738138    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.746524 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.746502    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"]
Nov 03 17:46:48.748207 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:46:48.748181    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8effaaf_1861_4257_8784_3c0df65f2d30.slice/crio-abe633393f6477ecff882feefe19a4c618271d92197690c3cffce1f865b5ac38 WatchSource:0}: Error finding container abe633393f6477ecff882feefe19a4c618271d92197690c3cffce1f865b5ac38: Status 404 returned error can't find the container with id abe633393f6477ecff882feefe19a4c618271d92197690c3cffce1f865b5ac38
Nov 03 17:46:48.750884 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.750852    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:48.750968 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.750927    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:48.751009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.750965    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:48.783127 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.783098    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.839680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.839637    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.839870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.839689    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.839870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.839723    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.839870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.839768    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840123 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840099    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840112    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840136    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840278 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840162    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840278 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840249    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840383 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840281    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xt9n\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840383 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840343    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840538 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840426    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840716 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840691    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.840814 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.840783    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.842349 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.842323    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.843361 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.843334    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.905043 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.905013    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.905273 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.905047    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xt9n\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:48.943925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.943891    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.962665 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:48.962633    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:49.112855 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.112823    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:49.289057 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.288963    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:49.317894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.317859    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"]
Nov 03 17:46:49.322871 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.322844    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.378328 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.378294    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-42c162-openshift-default-dockercfg-jf8vf\""
Nov 03 17:46:49.385136 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.385104    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"]
Nov 03 17:46:49.396757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.396693    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" event={"ID":"a8effaaf-1861-4257-8784-3c0df65f2d30","Type":"ContainerStarted","Data":"2ef3533508218569dc87873cc682e6c93bf8019c4417aac1d1c3f72b11fc7150"}
Nov 03 17:46:49.396910 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.396792    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" event={"ID":"a8effaaf-1861-4257-8784-3c0df65f2d30","Type":"ContainerStarted","Data":"abe633393f6477ecff882feefe19a4c618271d92197690c3cffce1f865b5ac38"}
Nov 03 17:46:49.414911 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.414879    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"]
Nov 03 17:46:49.424187 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:46:49.424159    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12538f55_31c0_492d_8087_360d54c7ff03.slice/crio-bd870d1767ab17194d6402abb4bd708a99d1f32903194ab9867ac6060bac2440 WatchSource:0}: Error finding container bd870d1767ab17194d6402abb4bd708a99d1f32903194ab9867ac6060bac2440: Status 404 returned error can't find the container with id bd870d1767ab17194d6402abb4bd708a99d1f32903194ab9867ac6060bac2440
Nov 03 17:46:49.427111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.427078    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:49.427216 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.427145    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:49.427216 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.427176    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:49.447315 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447288    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.447454 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447349    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.447513 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447488    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.447563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447544    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.447621 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447602    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.447723 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447705    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6qv\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.447926 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.447899    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.448025 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.448009    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.448091 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.448077    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.449596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.448688    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:49.549571 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549474    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6qv\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.549571 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549534    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549572    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549630    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549696    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549774    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549815    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549861    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.549889    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.550531 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.550264    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.551171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.551144    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.551355 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.551242    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.551482 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.551420    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.551612 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.551590    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.557783 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.556833    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.557783 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.557346    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.570397 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.570347    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podStartSLOduration=2.570327958 podStartE2EDuration="2.570327958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:49.568721466 +0000 UTC m=+745.579459565" watchObservedRunningTime="2026-11-03 17:46:49.570327958 +0000 UTC m=+745.581066095"
Nov 03 17:46:49.611819 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.611776    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:49.615216 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.615184    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6qv\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.619841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.619780    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.646859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.646827    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:49.769681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:49.769641    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:50.028273 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.028229    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:50.124620 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.124598    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"]
Nov 03 17:46:50.126798 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:46:50.126769    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52dc403_1aad_4f2a_b28c_8b14c1981c79.slice/crio-94a07a53a93dc5e186325cbec9c3cc97f848bfeb84de865ce36410b2eb7e57bb WatchSource:0}: Error finding container 94a07a53a93dc5e186325cbec9c3cc97f848bfeb84de865ce36410b2eb7e57bb: Status 404 returned error can't find the container with id 94a07a53a93dc5e186325cbec9c3cc97f848bfeb84de865ce36410b2eb7e57bb
Nov 03 17:46:50.129157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.129123    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:50.129237 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.129195    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:50.129237 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.129225    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:50.306190 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.306111    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:50.310908 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.310884    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:50.405192 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.405153    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" event={"ID":"e52dc403-1aad-4f2a-b28c-8b14c1981c79","Type":"ContainerStarted","Data":"23b9d0dfdb003ec1ed6bbe642be07455368ba018bd74e4815020aa4b30aec649"}
Nov 03 17:46:50.405389 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.405220    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" event={"ID":"e52dc403-1aad-4f2a-b28c-8b14c1981c79","Type":"ContainerStarted","Data":"94a07a53a93dc5e186325cbec9c3cc97f848bfeb84de865ce36410b2eb7e57bb"}
Nov 03 17:46:50.407045 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.407021    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" event={"ID":"12538f55-31c0-492d-8087-360d54c7ff03","Type":"ContainerStarted","Data":"c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"}
Nov 03 17:46:50.407166 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.407059    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" event={"ID":"12538f55-31c0-492d-8087-360d54c7ff03","Type":"ContainerStarted","Data":"bd870d1767ab17194d6402abb4bd708a99d1f32903194ab9867ac6060bac2440"}
Nov 03 17:46:50.407367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.407355    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:50.408367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.408337    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:46:50.572723 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.572623    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podStartSLOduration=2.572608103 podStartE2EDuration="2.572608103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:50.570355662 +0000 UTC m=+746.581093779" watchObservedRunningTime="2026-11-03 17:46:50.572608103 +0000 UTC m=+746.583346247"
Nov 03 17:46:50.647347 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.647314    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:50.828796 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.828668    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podStartSLOduration=1.8286535289999999 podStartE2EDuration="1.828653529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:50.826388974 +0000 UTC m=+746.837127046" watchObservedRunningTime="2026-11-03 17:46:50.828653529 +0000 UTC m=+746.839391656"
Nov 03 17:46:50.960384 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.960342    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"]
Nov 03 17:46:50.964380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.964302    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:50.964570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.964420    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:50.968628 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:50.968604    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:51.019902 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.019866    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-548ca1-openshift-default-dockercfg-z6wdl\""
Nov 03 17:46:51.030923 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.030888    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"]
Nov 03 17:46:51.065581 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.065548    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:51.074336 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074304    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074348    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074374    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074559 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074537    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074651 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074634    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074666    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074726    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl76\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074813    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.074862 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.074856    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.175959 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.175918    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.175990    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176022    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176060    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl76\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176304 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176274    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176362    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176414    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176451    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176476    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176640    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176794 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176771    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176872 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176844    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.176948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.176928    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.177113 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.177092    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.178485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.178464    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.178826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.178810    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.240372 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.240327    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.240544 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.240326    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl76\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.278995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.278951    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:51.376953 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.376910    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:51.414833 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.414798    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:51.415933 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.415911    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:46:51.654848 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.654820    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:51.715035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.715011    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"]
Nov 03 17:46:51.717555 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:46:51.717522    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d79d5ae_44d1_4160_ae68_3e42df3a51a9.slice/crio-524071fef650ed5da5c8e175b370e5db8a482a46bd6e27fb3a09e28b953a0a3f WatchSource:0}: Error finding container 524071fef650ed5da5c8e175b370e5db8a482a46bd6e27fb3a09e28b953a0a3f: Status 404 returned error can't find the container with id 524071fef650ed5da5c8e175b370e5db8a482a46bd6e27fb3a09e28b953a0a3f
Nov 03 17:46:51.720817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.720708    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:51.720916 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.720868    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:51.720916 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:51.720909    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:46:52.422114 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:52.422076    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" event={"ID":"9d79d5ae-44d1-4160-ae68-3e42df3a51a9","Type":"ContainerStarted","Data":"88f2c9d88aa365f556b095f0bf6828a03a5116194128512b1838002729f80ad2"}
Nov 03 17:46:52.422324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:52.422140    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" event={"ID":"9d79d5ae-44d1-4160-ae68-3e42df3a51a9","Type":"ContainerStarted","Data":"524071fef650ed5da5c8e175b370e5db8a482a46bd6e27fb3a09e28b953a0a3f"}
Nov 03 17:46:52.422722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:52.422702    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:52.423547 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:52.423527    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:46:52.547696 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:52.547648    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podStartSLOduration=2.547632945 podStartE2EDuration="2.547632945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:52.546004449 +0000 UTC m=+748.556742572" watchObservedRunningTime="2026-11-03 17:46:52.547632945 +0000 UTC m=+748.558371016"
Nov 03 17:46:52.927037 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:52.927003    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:53.122811 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.122771    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:53.279224 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.279142    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:53.284259 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.284230    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:53.285020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.285000    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:53.429444 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.429413    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:53.430393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.430376    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:46:53.848600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:53.848566    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:54.035175 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:54.035121    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:54.198790 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:54.198756    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:54.490709 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:54.490631    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:54.668580 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:54.668550    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:54.840915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:54.840834    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:55.108128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:55.108039    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:55.278708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:55.278674    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:55.439933 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:55.439903    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:55.788440 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:55.788341    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:55.951460 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:55.951424    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:56.112477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:56.112377    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:57.754248 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:57.754213    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:57.981706 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:57.981663    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:58.144110 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.144069    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:58.322785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.315565    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:58.472928 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.472840    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:58.784316 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.784239    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"]
Nov 03 17:46:58.788238 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.788216    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:58.848994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.848949    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"]
Nov 03 17:46:58.895563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.895527    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-07f297\""
Nov 03 17:46:58.895563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.895528    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-07f297-dockercfg-rz2fm\""
Nov 03 17:46:58.965072 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.965030    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:58.965259 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:58.965085    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmpx2\" (UniqueName: \"kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2\") pod \"payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:59.065934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.065833    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:59.065934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.065885    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmpx2\" (UniqueName: \"kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2\") pod \"payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:59.066548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.066522    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:59.129778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.129750    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmpx2\" (UniqueName: \"kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2\") pod \"payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:59.401901 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.401871    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:46:59.468310 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.468223    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:59.627474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.627441    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:59.781390 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.781368    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"]
Nov 03 17:46:59.783536 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:46:59.783503    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936f2578_8d84_4dab_b254_814e0fdc6ef0.slice/crio-5ae7534c76c724a303d2d5a7f4ccea283fe5c330c2e38d16dc3f6b522ec3cbf5 WatchSource:0}: Error finding container 5ae7534c76c724a303d2d5a7f4ccea283fe5c330c2e38d16dc3f6b522ec3cbf5: Status 404 returned error can't find the container with id 5ae7534c76c724a303d2d5a7f4ccea283fe5c330c2e38d16dc3f6b522ec3cbf5
Nov 03 17:46:59.792161 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:46:59.792136    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:00.484979 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:00.484946    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" event={"ID":"936f2578-8d84-4dab-b254-814e0fdc6ef0","Type":"ContainerStarted","Data":"81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c"}
Nov 03 17:47:00.485221 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:00.484994    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" event={"ID":"936f2578-8d84-4dab-b254-814e0fdc6ef0","Type":"ContainerStarted","Data":"5ae7534c76c724a303d2d5a7f4ccea283fe5c330c2e38d16dc3f6b522ec3cbf5"}
Nov 03 17:47:00.485221 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:00.485047    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:47:00.606564 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:00.606510    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" podStartSLOduration=2.60649751 podStartE2EDuration="2.60649751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:00.605614784 +0000 UTC m=+756.616352879" watchObservedRunningTime="2026-11-03 17:47:00.60649751 +0000 UTC m=+756.617235629"
Nov 03 17:47:03.250539 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.250494    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:03.426183 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.426143    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:03.667008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.666967    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"]
Nov 03 17:47:03.670981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.670957    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.718934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.718898    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpgn\" (UniqueName: \"kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn\") pod \"payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.719108 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.719025    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.732148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.732115    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"]
Nov 03 17:47:03.777894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.777861    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-f7d463-dockercfg-8rrjp\""
Nov 03 17:47:03.778302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.778279    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-f7d463\""
Nov 03 17:47:03.810637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.810599    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:03.819961 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.819929    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.820130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.820064    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpgn\" (UniqueName: \"kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn\") pod \"payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.820755 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.820710    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.830174 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.830149    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"]
Nov 03 17:47:03.834175 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.834155    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:03.868423 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.868386    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"]
Nov 03 17:47:03.883904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.883876    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpgn\" (UniqueName: \"kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn\") pod \"payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:03.921167 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.921079    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd285\" (UniqueName: \"kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285\") pod \"payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:03.921167 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.921144    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:03.985073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:03.985030    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:04.002274 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.002242    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:04.022602 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.022569    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd285\" (UniqueName: \"kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285\") pod \"payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:04.022772 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.022636    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:04.023345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.023327    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:04.098838 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.098800    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd285\" (UniqueName: \"kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285\") pod \"payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:04.147614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.147577    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:04.175163 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.175089    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:04.365447 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.365418    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"]
Nov 03 17:47:04.366824 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:04.366797    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf037b0ba_514f_4ca7_a1b7_5943f29ae94d.slice/crio-a8f512c9b3db9848b6922bbad003e45489c44df89c28aceabc459fd580e2500e WatchSource:0}: Error finding container a8f512c9b3db9848b6922bbad003e45489c44df89c28aceabc459fd580e2500e: Status 404 returned error can't find the container with id a8f512c9b3db9848b6922bbad003e45489c44df89c28aceabc459fd580e2500e
Nov 03 17:47:04.519772 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.515494    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" event={"ID":"f037b0ba-514f-4ca7-a1b7-5943f29ae94d","Type":"ContainerStarted","Data":"f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d"}
Nov 03 17:47:04.519772 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.515563    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" event={"ID":"f037b0ba-514f-4ca7-a1b7-5943f29ae94d","Type":"ContainerStarted","Data":"a8f512c9b3db9848b6922bbad003e45489c44df89c28aceabc459fd580e2500e"}
Nov 03 17:47:04.519772 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.516522    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:04.531506 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.531481    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"]
Nov 03 17:47:04.532494 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:04.532471    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4d401c_e67d_444d_9391_0d36da0c9db4.slice/crio-23f04bebe3e04bb1a37a91bab60034811bf8b140e50580d7262edc28b7bf8370 WatchSource:0}: Error finding container 23f04bebe3e04bb1a37a91bab60034811bf8b140e50580d7262edc28b7bf8370: Status 404 returned error can't find the container with id 23f04bebe3e04bb1a37a91bab60034811bf8b140e50580d7262edc28b7bf8370
Nov 03 17:47:04.639040 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:04.638988    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" podStartSLOduration=1.638973341 podStartE2EDuration="1.638973341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:04.636445843 +0000 UTC m=+760.647183940" watchObservedRunningTime="2026-11-03 17:47:04.638973341 +0000 UTC m=+760.649711412"
Nov 03 17:47:05.024886 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.024854    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:05.246438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.246403    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:05.404451 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.404412    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:05.523357 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.523318    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" event={"ID":"0c4d401c-e67d-444d-9391-0d36da0c9db4","Type":"ContainerStarted","Data":"4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3"}
Nov 03 17:47:05.523555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.523385    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" event={"ID":"0c4d401c-e67d-444d-9391-0d36da0c9db4","Type":"ContainerStarted","Data":"23f04bebe3e04bb1a37a91bab60034811bf8b140e50580d7262edc28b7bf8370"}
Nov 03 17:47:05.523618 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.523582    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:05.645791 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.645724    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" podStartSLOduration=2.645709599 podStartE2EDuration="2.645709599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:05.643807169 +0000 UTC m=+761.654545244" watchObservedRunningTime="2026-11-03 17:47:05.645709599 +0000 UTC m=+761.656447670"
Nov 03 17:47:05.906388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:05.906353    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:06.086955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:06.086921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:08.037450 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.037417    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:47:08.471163 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.471126    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:08.651594 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.651567    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:08.878266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.878188    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"]
Nov 03 17:47:08.882082 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.882064    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:08.942748 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.942692    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"]
Nov 03 17:47:08.977069 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.977026    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kjb\" (UniqueName: \"kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb\") pod \"payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:08.977262 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.977095    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:08.989316 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.989257    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-42c162-dockercfg-k47lh\""
Nov 03 17:47:08.993375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:08.993306    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-42c162\""
Nov 03 17:47:09.078026 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:09.077974    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kjb\" (UniqueName: \"kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb\") pod \"payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:09.078418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:09.078045    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:09.078826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:09.078804    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:09.142473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:09.142438    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kjb\" (UniqueName: \"kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb\") pod \"payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:09.196276 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:09.196239    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:09.581810 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:09.581670    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"]
Nov 03 17:47:09.588252 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:09.588222    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c40819a_98af_412e_8542_ccf40133be24.slice/crio-8f4acac1b3a03fcb37a1bcb8a624dda4aa4e5447ebf1a63c0dce6cc19ac95d90 WatchSource:0}: Error finding container 8f4acac1b3a03fcb37a1bcb8a624dda4aa4e5447ebf1a63c0dce6cc19ac95d90: Status 404 returned error can't find the container with id 8f4acac1b3a03fcb37a1bcb8a624dda4aa4e5447ebf1a63c0dce6cc19ac95d90
Nov 03 17:47:10.338974 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:10.338933    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:47:10.351059 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.351029    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"]
Nov 03 17:47:10.355275 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.355254    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.394366 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.394332    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7hx\" (UniqueName: \"kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.394542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.394386    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.394542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.394420    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.394656 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.394549    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.410195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.410165    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-42c162-serving-cert\""
Nov 03 17:47:10.462045 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.462011    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"]
Nov 03 17:47:10.496179 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.496136    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.496375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.496250    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.496375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.496363    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7hx\" (UniqueName: \"kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.496496 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.496417    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.496585 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:10.496568    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-serving-cert: secret "maas-api-e2e-worker-w2-42c162-serving-cert" not found
Nov 03 17:47:10.496672 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:10.496662    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls podName:e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a nodeName:}" failed. No retries permitted until 2026-11-03 17:47:10.996622617 +0000 UTC m=+767.007360682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls") pod "maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" (UID: "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a") : secret "maas-api-e2e-worker-w2-42c162-serving-cert" not found
Nov 03 17:47:10.497257 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.497233    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.499328 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.499301    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.556211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.556173    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" event={"ID":"7c40819a-98af-412e-8542-ccf40133be24","Type":"ContainerStarted","Data":"d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3"}
Nov 03 17:47:10.556400 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.556237    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" event={"ID":"7c40819a-98af-412e-8542-ccf40133be24","Type":"ContainerStarted","Data":"8f4acac1b3a03fcb37a1bcb8a624dda4aa4e5447ebf1a63c0dce6cc19ac95d90"}
Nov 03 17:47:10.556400 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.556267    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:10.561554 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.561524    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:10.562346 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.562325    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7hx\" (UniqueName: \"kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:10.678157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.678106    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" podStartSLOduration=2.678090388 podStartE2EDuration="2.678090388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:10.676094376 +0000 UTC m=+766.686832461" watchObservedRunningTime="2026-11-03 17:47:10.678090388 +0000 UTC m=+766.688828480"
Nov 03 17:47:10.766881 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.766847    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:10.925796 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:10.925761    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:11.003161 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:11.003076    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:11.005647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:11.005623    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:11.268622 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:11.268532    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:11.494227 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:11.494195    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 17:47:11.657680 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:11.657633    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a4c0d4_4620_4d1c_874a_ac8f3ed84d2a.slice/crio-ff3bead07a2bc45ff20b12166da0b86a58d237a19f021a5453247cad04c9df9d WatchSource:0}: Error finding container ff3bead07a2bc45ff20b12166da0b86a58d237a19f021a5453247cad04c9df9d: Status 404 returned error can't find the container with id ff3bead07a2bc45ff20b12166da0b86a58d237a19f021a5453247cad04c9df9d
Nov 03 17:47:11.668027 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:11.667998    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"]
Nov 03 17:47:12.570584 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:12.570539    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" event={"ID":"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a","Type":"ContainerStarted","Data":"8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185"}
Nov 03 17:47:12.571116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:12.570610    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" event={"ID":"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a","Type":"ContainerStarted","Data":"ff3bead07a2bc45ff20b12166da0b86a58d237a19f021a5453247cad04c9df9d"}
Nov 03 17:47:12.571116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:12.570723    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:12.728475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:12.728362    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" podStartSLOduration=1.950804822 podStartE2EDuration="2.728339302s" totalImagesPullingTime="777.53448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:12.725857455 +0000 UTC m=+768.736595555" watchObservedRunningTime="2026-11-03 17:47:12.728339302 +0000 UTC m=+768.739077374"
Nov 03 17:47:13.437702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:13.437661    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:13.620845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:13.620797    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:14.000486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.000442    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"]
Nov 03 17:47:14.005920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.005889    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.063627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.063594    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c76280\""
Nov 03 17:47:14.067955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.067925    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"]
Nov 03 17:47:14.114353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.114321    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c76280-dockercfg-ll68d\""
Nov 03 17:47:14.144880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.144847    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrv6\" (UniqueName: \"kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6\") pod \"payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.145074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.145049    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.245898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.245863    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.246087 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.246009    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrv6\" (UniqueName: \"kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6\") pod \"payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.246673 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.246644    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.309836 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.309754    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrv6\" (UniqueName: \"kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6\") pod \"payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.322645 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.322614    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:14.714245 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:14.714216    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"]
Nov 03 17:47:14.715574 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:14.715544    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95793ffb_cf3e_4cb2_8a0e_b72aca7753c8.slice/crio-88e18ba68f9bdf7f343f3999369ea4d92369341804e71f5d47967c138fb90e6d WatchSource:0}: Error finding container 88e18ba68f9bdf7f343f3999369ea4d92369341804e71f5d47967c138fb90e6d: Status 404 returned error can't find the container with id 88e18ba68f9bdf7f343f3999369ea4d92369341804e71f5d47967c138fb90e6d
Nov 03 17:47:15.590637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:15.590603    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" event={"ID":"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8","Type":"ContainerStarted","Data":"968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef"}
Nov 03 17:47:15.590830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:15.590772    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:15.590830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:15.590798    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" event={"ID":"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8","Type":"ContainerStarted","Data":"88e18ba68f9bdf7f343f3999369ea4d92369341804e71f5d47967c138fb90e6d"}
Nov 03 17:47:15.712024 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:15.711965    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" podStartSLOduration=2.711948445 podStartE2EDuration="2.711948445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:15.710759359 +0000 UTC m=+771.721497431" watchObservedRunningTime="2026-11-03 17:47:15.711948445 +0000 UTC m=+771.722686517"
Nov 03 17:47:16.011439 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:16.011402    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:16.170614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:16.170583    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:16.327902 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:16.327811    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:16.531799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:16.531766    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 17:47:16.531986 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:16.531970    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 17:47:17.353222 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:17.353178    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:47:18.583207 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:18.583177    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 17:47:19.353763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:19.352598    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:21.089873 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:21.089818    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:21.249291 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:21.249264    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:21.437630 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:21.437592    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:21.573296 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:21.573267    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 17:47:21.608828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:21.608723    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:26.066745 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.066694    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"]
Nov 03 17:47:26.080098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.080062    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.133662 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.133619    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"]
Nov 03 17:47:26.178431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.178395    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-07f297-serving-cert\""
Nov 03 17:47:26.253092 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.253057    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.253308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.253118    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.253308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.253240    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.253308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.253286    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftd8\" (UniqueName: \"kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.262145 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.262114    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:26.354839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.354744    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.354839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.354825    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.355094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.354855    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftd8\" (UniqueName: \"kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.355094 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:26.354977    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-serving-cert: secret "maas-api-e2e-worker-w0-07f297-serving-cert" not found
Nov 03 17:47:26.355094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.355011    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.355094 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:26.355052    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls podName:ece16545-f72f-45ac-bfae-2723829f1d31 nodeName:}" failed. No retries permitted until 2026-11-03 17:47:26.855031778 +0000 UTC m=+782.865769848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls") pod "maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" (UID: "ece16545-f72f-45ac-bfae-2723829f1d31") : secret "maas-api-e2e-worker-w0-07f297-serving-cert" not found
Nov 03 17:47:26.355722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.355700    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.358710 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.358685    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.419611 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.419571    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftd8\" (UniqueName: \"kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.430558 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.430523    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:26.618103 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.618021    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 17:47:26.661104 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.661050    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:26.865958 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.865922    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.868666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.868595    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-07f297-789b84544-j6jqt\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:26.995144 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:26.995102    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:27.442067 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:27.442029    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece16545_f72f_45ac_bfae_2723829f1d31.slice/crio-9b089884d7f8f0b0b91013c27b8c21b4e216e15421188f9b9a5ac475ab7d22b5 WatchSource:0}: Error finding container 9b089884d7f8f0b0b91013c27b8c21b4e216e15421188f9b9a5ac475ab7d22b5: Status 404 returned error can't find the container with id 9b089884d7f8f0b0b91013c27b8c21b4e216e15421188f9b9a5ac475ab7d22b5
Nov 03 17:47:27.442530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.442139    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"]
Nov 03 17:47:27.716239 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.716205    5838 generic.go:356] "Generic (PLEG): container finished" podID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerID="23b9d0dfdb003ec1ed6bbe642be07455368ba018bd74e4815020aa4b30aec649" exitCode=0
Nov 03 17:47:27.716437 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.716281    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" event={"ID":"e52dc403-1aad-4f2a-b28c-8b14c1981c79","Type":"ContainerDied","Data":"23b9d0dfdb003ec1ed6bbe642be07455368ba018bd74e4815020aa4b30aec649"}
Nov 03 17:47:27.716849 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.716825    5838 scope.go:122] "RemoveContainer" containerID="23b9d0dfdb003ec1ed6bbe642be07455368ba018bd74e4815020aa4b30aec649"
Nov 03 17:47:27.718050 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.718029    5838 generic.go:356] "Generic (PLEG): container finished" podID="12538f55-31c0-492d-8087-360d54c7ff03" containerID="c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e" exitCode=0
Nov 03 17:47:27.718141 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.718101    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" event={"ID":"12538f55-31c0-492d-8087-360d54c7ff03","Type":"ContainerDied","Data":"c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"}
Nov 03 17:47:27.718290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.718257    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:27.718356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.718336    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:27.718394 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.718379    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:27.718458 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.718438    5838 scope.go:122] "RemoveContainer" containerID="c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"
Nov 03 17:47:27.719459 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.719433    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" event={"ID":"ece16545-f72f-45ac-bfae-2723829f1d31","Type":"ContainerStarted","Data":"9b089884d7f8f0b0b91013c27b8c21b4e216e15421188f9b9a5ac475ab7d22b5"}
Nov 03 17:47:27.719720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.719698    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:27.719800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.719785    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:27.719836 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:27.719821    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:28.759390 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.759318    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" event={"ID":"e52dc403-1aad-4f2a-b28c-8b14c1981c79","Type":"ContainerStarted","Data":"dda9bba30058ec149df47aed11648e3dc62b4a3391bb01b98e6cde9531358e10"}
Nov 03 17:47:28.761910 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.761883    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" event={"ID":"12538f55-31c0-492d-8087-360d54c7ff03","Type":"ContainerStarted","Data":"7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b"}
Nov 03 17:47:28.763441 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.763419    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" event={"ID":"ece16545-f72f-45ac-bfae-2723829f1d31","Type":"ContainerStarted","Data":"41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108"}
Nov 03 17:47:28.763641 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.763599    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:28.963650 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.963613    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:47:28.964616 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.963904    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 17:47:28.964616 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:28.963977    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 17:47:29.223758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.223353    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" podStartSLOduration=2.391634409 podStartE2EDuration="3.223330642s" totalImagesPullingTime="831.696233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:29.044995331 +0000 UTC m=+785.055733425" watchObservedRunningTime="2026-11-03 17:47:29.223330642 +0000 UTC m=+785.234068714"
Nov 03 17:47:29.650760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.650178    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:47:29.650760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.650315    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:47:29.650760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.650362    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 17:47:29.674235 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.674195    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:29.963188 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.963101    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 17:47:29.963188 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:29.963150    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 17:47:30.648221 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:30.648192    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:47:30.648508 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:30.648479    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 17:47:30.963302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:30.963201    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 17:47:30.963302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:30.963254    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 17:47:31.647704 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:31.647664    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:47:31.647918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:31.647725    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 17:47:31.845760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:31.845664    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:31.964151 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:31.964050    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 17:47:31.964151 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:31.964108    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 17:47:32.647376 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:32.647341    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:47:32.647567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:32.647403    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 17:47:32.963811 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:32.963701    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 17:47:32.963811 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:32.963778    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 17:47:33.650220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:33.650179    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:47:33.650903 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:33.650869    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 17:47:33.963879 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:33.963790    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 17:47:33.963879 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:33.963842    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 17:47:34.652670 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.652638    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:47:34.653116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.653090    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:47:34.653930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.653904    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 17:47:34.775826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.775799    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 17:47:34.805256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.805221    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:34.968543 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.968460    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:47:34.968780 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.968764    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:47:34.969585 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:34.969565    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 17:47:36.811054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:36.810990    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:36.927923 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:36.927890    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:47:38.278258 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:38.278224    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:38.930793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:38.930724    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:39.090715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:39.090675    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:39.250821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:39.250712    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:40.349035 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:40.348994    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:47:42.424537 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:42.424462    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:42.624490 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:42.624445    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:42.777033 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:42.776935    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:43.073144 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:43.073052    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:44.665810 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:44.665767    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:45.282043 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:45.281993    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:45.448268 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:45.448232    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:45.626149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:45.626057    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:47.891211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.891173    5838 generic.go:356] "Generic (PLEG): container finished" podID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerID="2ef3533508218569dc87873cc682e6c93bf8019c4417aac1d1c3f72b11fc7150" exitCode=0
Nov 03 17:47:47.891706 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.891247    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" event={"ID":"a8effaaf-1861-4257-8784-3c0df65f2d30","Type":"ContainerDied","Data":"2ef3533508218569dc87873cc682e6c93bf8019c4417aac1d1c3f72b11fc7150"}
Nov 03 17:47:47.891706 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.891643    5838 scope.go:122] "RemoveContainer" containerID="2ef3533508218569dc87873cc682e6c93bf8019c4417aac1d1c3f72b11fc7150"
Nov 03 17:47:47.893046 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.893014    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:47.893160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.893103    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:47.893160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.893152    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:47:47.998668 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:47.998641    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"]
Nov 03 17:47:48.001633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.001605    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.042108 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.042075    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"]
Nov 03 17:47:48.057239 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.057208    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-548ca1-serving-cert\""
Nov 03 17:47:48.095913 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.095877    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ld4\" (UniqueName: \"kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.096149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.095962    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.096149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.096063    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.096149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.096093    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.197799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.197687    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.197799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.197777    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.198048 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.197926    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ld4\" (UniqueName: \"kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.198048 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.197995    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.198161 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:48.198128    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-serving-cert: secret "maas-api-e2e-worker-w5-548ca1-serving-cert" not found
Nov 03 17:47:48.198210 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:47:48.198195    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls podName:f754563c-befa-48c9-89a5-60a02450fd03 nodeName:}" failed. No retries permitted until 2026-11-03 17:47:48.698173617 +0000 UTC m=+804.708911685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls") pod "maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" (UID: "f754563c-befa-48c9-89a5-60a02450fd03") : secret "maas-api-e2e-worker-w5-548ca1-serving-cert" not found
Nov 03 17:47:48.198607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.198579    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.200412 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.200389    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.242506 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.242456    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:48.265678 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.265648    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ld4\" (UniqueName: \"kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.400756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.400313    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:48.570770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.570668    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:48.711755 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.707549    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.724493 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.724420    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\") pod \"maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.918753 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.918577    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:48.925081 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:48.922925    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" event={"ID":"a8effaaf-1861-4257-8784-3c0df65f2d30","Type":"ContainerStarted","Data":"b9382ecdb8ec619c6d05f36aa8de892e3a31f7179f573acaa71960bb6e436c11"}
Nov 03 17:47:49.312820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:49.312098    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:47:49.314761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:49.313768    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 17:47:49.314761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:49.313830    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 17:47:49.355382 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:49.355349    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"]
Nov 03 17:47:49.356832 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:47:49.356791    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf754563c_befa_48c9_89a5_60a02450fd03.slice/crio-4081aef76fd0bf8043d7a8a8408849aee15bc7808deda7676b5a7cfada0fe278 WatchSource:0}: Error finding container 4081aef76fd0bf8043d7a8a8408849aee15bc7808deda7676b5a7cfada0fe278: Status 404 returned error can't find the container with id 4081aef76fd0bf8043d7a8a8408849aee15bc7808deda7676b5a7cfada0fe278
Nov 03 17:47:49.931061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:49.931027    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" event={"ID":"f754563c-befa-48c9-89a5-60a02450fd03","Type":"ContainerStarted","Data":"4081aef76fd0bf8043d7a8a8408849aee15bc7808deda7676b5a7cfada0fe278"}
Nov 03 17:47:50.312758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:50.311315    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 17:47:50.312758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:50.311394    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 17:47:50.414760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:50.409868    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:47:50.939362 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:50.939326    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" event={"ID":"f754563c-befa-48c9-89a5-60a02450fd03","Type":"ContainerStarted","Data":"29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234"}
Nov 03 17:47:50.939883 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:50.939472    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:51.066073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:51.065998    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" podStartSLOduration=3.198309304 podStartE2EDuration="4.065974238s" totalImagesPullingTime="867.664934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:51.06455938 +0000 UTC m=+807.075297491" watchObservedRunningTime="2026-11-03 17:47:51.065974238 +0000 UTC m=+807.076712311"
Nov 03 17:47:51.320816 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:51.320197    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 17:47:51.320816 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:51.320258    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 17:47:52.309770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:52.309157    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 17:47:52.309770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:52.309221    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 17:47:53.059583 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:53.059550    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:47:53.307717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:53.307230    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 17:47:53.307717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:53.307292    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 17:47:54.133008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:54.132967    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:54.310502 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:54.310468    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:47:54.311418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:54.311396    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 17:47:56.952747 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:56.952705    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 17:47:59.521134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:47:59.521096    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:00.137948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:00.137901    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:00.791698 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:00.791666    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:00.953987 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:00.953946    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:01.113034 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:01.112940    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:04.255852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:04.248630    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:48:05.363099 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:05.363063    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:06.427810 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:06.427770    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:10.340261 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:48:10.340214    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:48:10.548164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:10.548013    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:14.053460 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:14.052152    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:14.113625 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:14.113587    5838 generic.go:356] "Generic (PLEG): container finished" podID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerID="88f2c9d88aa365f556b095f0bf6828a03a5116194128512b1838002729f80ad2" exitCode=0
Nov 03 17:48:14.117155 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:14.117120    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" event={"ID":"9d79d5ae-44d1-4160-ae68-3e42df3a51a9","Type":"ContainerDied","Data":"88f2c9d88aa365f556b095f0bf6828a03a5116194128512b1838002729f80ad2"}
Nov 03 17:48:15.119936 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:15.119900    5838 scope.go:122] "RemoveContainer" containerID="88f2c9d88aa365f556b095f0bf6828a03a5116194128512b1838002729f80ad2"
Nov 03 17:48:15.121458 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:15.121426    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:48:15.121521 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:15.121504    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:48:15.121556 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:15.121536    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:48:15.777919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:15.777883    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:15.934574 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:15.934538    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:16.132364 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.132271    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" event={"ID":"9d79d5ae-44d1-4160-ae68-3e42df3a51a9","Type":"ContainerStarted","Data":"e04a4894ffca0b36494777caf4d7b811bf3f01c21a3eb5be88068937dcdc0adf"}
Nov 03 17:48:16.279321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.279280    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:48:16.279522 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.279500    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 17:48:16.279582 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.279563    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 17:48:16.427177 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.427143    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:16.589922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.589868    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:16.749994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:16.749888    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:17.279531 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:17.279500    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 17:48:17.279971 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:17.279552    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 17:48:18.280248 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:18.280215    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 17:48:18.280680 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:18.280282    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 17:48:19.280321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:19.280288    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 17:48:19.280754 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:19.280365    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 17:48:19.463006 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:19.462973    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:20.117301 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:20.117264    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:20.279719 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:20.279677    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 17:48:20.279970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:20.279778    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 17:48:20.284265 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:20.284230    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:20.443060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:20.443019    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:21.211709 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:21.211661    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:21.280149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:21.280108    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 17:48:21.280332 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:21.280177    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 17:48:22.286202 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:22.286168    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:48:22.287579 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:22.287551    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:48:22.288596 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:22.288576    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 17:48:23.027647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:23.027606    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:48:26.296125 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:26.296061    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:29.864279 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:29.864245    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:31.403799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:31.403764    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:35.148105 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:35.148060    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:36.417322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:36.417276    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:36.811613 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:36.811515    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:37.132116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:37.132025    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:37.298744 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:37.298685    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:37.461335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:37.461298    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:37.962462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:37.962398    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:38.350436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:38.350350    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:48:38.687456 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:38.687407    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:38.858485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:38.858453    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:40.339121 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:48:40.339083    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:48:42.301981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:42.301940    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:42.712573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:42.712537    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:47.434298 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:47.434259    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:52.498845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:52.498809    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:54.309451 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:54.309411    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:57.592897 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:48:57.592858    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:00.598360 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:00.598324    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:49:02.682788 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:02.682752    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:06.828036 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:06.827994    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:07.816375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:07.816331    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:10.340875 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:49:10.340835    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:49:13.008523 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:13.008467    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:18.094637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:18.094596    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:18.308101 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:18.308062    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:23.196504 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:23.196465    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:23.359349 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:23.359303    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:24.274044 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:24.274014    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:49:24.279133 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:24.279110    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:49:28.298455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:28.298409    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:29.140524 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:29.140490    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:49:29.700090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:29.700052    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:33.414441 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:33.414342    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:35.567348 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:35.567306    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:38.487121 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:38.487075    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:40.339155 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:49:40.339115    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:49:43.515335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:43.515292    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:48.492281 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:48.492247    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:48.652898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:48.652863    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:53.610614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:53.610579    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:54.675212 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:54.675177    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:49:58.699126 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:49:58.699091    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:02.989142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:02.989107    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:03.817804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:03.817759    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:08.997844 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:08.997807    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:10.339438 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:50:10.339384    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:50:14.014437 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:14.014398    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:15.878441 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:15.878408    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:50:19.129277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:19.129207    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:24.186514 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.186484    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:24.391355 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.391321    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"]
Nov 03 17:50:24.395443 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.395421    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.465285 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.465195    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"]
Nov 03 17:50:24.487034 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.486994    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:24.499999 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.499963    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"kube-root-ca.crt\""
Nov 03 17:50:24.499999 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.499978    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"openshift-service-ca.crt\""
Nov 03 17:50:24.500393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.500378    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"premium-w0-07f297-kserve-self-signed-certs\""
Nov 03 17:50:24.502893 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.502868    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"default-dockercfg-dn566\""
Nov 03 17:50:24.509960 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.509939    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.510045 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.509999    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgzp\" (UniqueName: \"kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.510123 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.510103    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.510197 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.510177    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.510271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.510250    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.510361 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.510347    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.611882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.611839    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgzp\" (UniqueName: \"kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.611899    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.611947    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.611984    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612098 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.612065    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612316 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.612138    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.612346    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612448 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.612427    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.612505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.612473    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.614592 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.614560    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.614902 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.614883    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.676384 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.676337    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgzp\" (UniqueName: \"kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp\") pod \"premium-w0-07f297-kserve-ddf4b9fbf-kzm8m\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") " pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:24.708248 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:24.708207    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:25.118007 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:25.117978    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"]
Nov 03 17:50:25.120080 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:50:25.120049    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e82a454_4d93_496d_8557_2d434bf780f9.slice/crio-e5df06eaf4e0e8a53a733147c12f599a89beb0572fb02673ef3f5bda38b20a21 WatchSource:0}: Error finding container e5df06eaf4e0e8a53a733147c12f599a89beb0572fb02673ef3f5bda38b20a21: Status 404 returned error can't find the container with id e5df06eaf4e0e8a53a733147c12f599a89beb0572fb02673ef3f5bda38b20a21
Nov 03 17:50:26.080550 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:26.080509    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" event={"ID":"8e82a454-4d93-496d-8557-2d434bf780f9","Type":"ContainerStarted","Data":"0f30f81f0d04dcbba81864816e775abdce4a40f819e97e3ce6f05b1c4a0ae418"}
Nov 03 17:50:26.080550 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:26.080560    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" event={"ID":"8e82a454-4d93-496d-8557-2d434bf780f9","Type":"ContainerStarted","Data":"e5df06eaf4e0e8a53a733147c12f599a89beb0572fb02673ef3f5bda38b20a21"}
Nov 03 17:50:26.202906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:26.202848    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podStartSLOduration=2.202833197 podStartE2EDuration="2.202833197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:26.201017343 +0000 UTC m=+962.211755436" watchObservedRunningTime="2026-11-03 17:50:26.202833197 +0000 UTC m=+962.213571269"
Nov 03 17:50:26.708717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:26.708678    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:29.295271 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:29.295237    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:33.745028 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:33.744985    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:34.437051 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:34.437017    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:34.709384 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:34.709287    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:34.728505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:34.728473    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:35.166302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:35.166262    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:50:35.851127 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:35.851091    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:39.491290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:39.491247    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:40.339582 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:50:40.339537    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:50:41.004324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:41.004282    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:41.178034 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:41.177996    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:44.611826 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:44.611791    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:45.932760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:45.932690    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:47.637396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:47.637364    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:50:49.795196 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:49.795162    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:51.091509 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.091464    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"]
Nov 03 17:50:51.096042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.096009    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.202006 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.201969    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c76280-openshift-default-dockercfg-mtxr6\""
Nov 03 17:50:51.206715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.206659    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"]
Nov 03 17:50:51.214457 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214414    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214534    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214567    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214615    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214635    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214755    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214802    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214852    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29p4\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.214918 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.214904    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.315868 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.315834    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.316079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.315899    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.316079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.315937    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.316079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.315961    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.316079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316002    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.316079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316018    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.316079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316061    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316107    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316152    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29p4\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316395    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316674    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316795    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.316936    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.317240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.317223    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.319708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.319680    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.319872 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.319853    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.380614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.380530    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.380804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.380715    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29p4\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.413950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.413918    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:51.852956 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:50:51.852921    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77397099_c52d_41db_b1ab_811d8b439547.slice/crio-91eef89ecc76884cc75d6c97de6ab05d9da51aaa48da98c5cb695aad30f831c4 WatchSource:0}: Error finding container 91eef89ecc76884cc75d6c97de6ab05d9da51aaa48da98c5cb695aad30f831c4: Status 404 returned error can't find the container with id 91eef89ecc76884cc75d6c97de6ab05d9da51aaa48da98c5cb695aad30f831c4
Nov 03 17:50:51.853110 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.852982    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"]
Nov 03 17:50:51.855849 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.855807    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:50:51.855956 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.855901    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:50:51.855956 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:51.855946    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:50:52.286064 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:52.286030    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" event={"ID":"77397099-c52d-41db-b1ab-811d8b439547","Type":"ContainerStarted","Data":"9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d"}
Nov 03 17:50:52.286471 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:52.286087    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" event={"ID":"77397099-c52d-41db-b1ab-811d8b439547","Type":"ContainerStarted","Data":"91eef89ecc76884cc75d6c97de6ab05d9da51aaa48da98c5cb695aad30f831c4"}
Nov 03 17:50:52.414140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:52.414082    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podStartSLOduration=1.414066805 podStartE2EDuration="1.414066805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:52.410977774 +0000 UTC m=+988.421715846" watchObservedRunningTime="2026-11-03 17:50:52.414066805 +0000 UTC m=+988.424804954"
Nov 03 17:50:52.414806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:52.414782    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:52.416674 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:52.416647    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:52.416811 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:52.416692    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:53.416240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:53.416202    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:53.416897 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:53.416863    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:54.419758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:54.419162    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:54.419758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:54.419229    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:54.922393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:54.922348    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:55.417757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:55.417342    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:55.417757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:55.417403    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:56.419758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:56.419185    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:56.419758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:56.419252    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:57.416250 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:57.416206    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:57.416578 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:57.416544    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:58.414457 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:58.414419    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:50:58.414937 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:58.414480    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 17:50:59.419137 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:59.419106    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:59.419538 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:59.419385    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:50:59.420178 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:50:59.420162    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 17:51:00.009988 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:00.009956    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:03.975226 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:03.975129    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:05.124688 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:05.124625    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:06.045944 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:06.045908    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"]
Nov 03 17:51:06.046222 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:06.046196    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy" containerID="cri-o://601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b" gracePeriod=30
Nov 03 17:51:10.204746 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:10.204688    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:10.339920 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:51:10.339875    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:51:11.331698 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.331623    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:51:11.361413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361374    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.361413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361411    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.361676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361435    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.361676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361459    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.361676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361537    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.361676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361581    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.361676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.361646    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.362151 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362072    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:51:11.362151 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362113    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:51:11.362519 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362159    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:11.362519 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362297    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.362519 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362314    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4f6054-05f5-41c5-848f-b245cbff25d3-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.362519 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362327    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.362761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.362098    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:51:11.364170 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.364133    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:51:11.364170 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.364153    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:51:11.364306 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.364170    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:11.450898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.450861    5838 generic.go:356] "Generic (PLEG): container finished" podID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerID="601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b" exitCode=0
Nov 03 17:51:11.451058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.450925    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"
Nov 03 17:51:11.451058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.450941    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" event={"ID":"3f4f6054-05f5-41c5-848f-b245cbff25d3","Type":"ContainerDied","Data":"601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b"}
Nov 03 17:51:11.451058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.451004    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l" event={"ID":"3f4f6054-05f5-41c5-848f-b245cbff25d3","Type":"ContainerDied","Data":"5c566f83c4974016427d393ebd6d59909b68d726f7ee4bd0eaddb443bf839859"}
Nov 03 17:51:11.451058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.451037    5838 scope.go:122] "RemoveContainer" containerID="601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b"
Nov 03 17:51:11.461978 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.461953    5838 scope.go:122] "RemoveContainer" containerID="cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"
Nov 03 17:51:11.462854 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.462832    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.462925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.462870    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk\") pod \"3f4f6054-05f5-41c5-848f-b245cbff25d3\" (UID: \"3f4f6054-05f5-41c5-848f-b245cbff25d3\") "
Nov 03 17:51:11.463196 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.463168    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:51:11.463319 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.463306    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.463374 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.463323    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.463374 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.463333    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.463374 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.463345    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.463374 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.463361    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4f6054-05f5-41c5-848f-b245cbff25d3-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.465219 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.465194    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk" pod "3f4f6054-05f5-41c5-848f-b245cbff25d3" (UID: "3f4f6054-05f5-41c5-848f-b245cbff25d3"). InnerVolumeSpecName "kube-api-access-8lkgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:11.471875 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.471850    5838 scope.go:122] "RemoveContainer" containerID="601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b"
Nov 03 17:51:11.472232 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:51:11.472211    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b\": container with ID starting with 601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b not found: ID does not exist" containerID="601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b"
Nov 03 17:51:11.472311 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.472246    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b"} err="failed to get container status \"601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b\": rpc error: code = NotFound desc = could not find container \"601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b\": container with ID starting with 601bac979015c80c46acd2e284a1a4bacd3e98b39ec9a82c446aac4770acb23b not found: ID does not exist"
Nov 03 17:51:11.472311 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.472272    5838 scope.go:122] "RemoveContainer" containerID="cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"
Nov 03 17:51:11.472558 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:51:11.472540    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411\": container with ID starting with cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411 not found: ID does not exist" containerID="cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"
Nov 03 17:51:11.472601 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.472566    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411"} err="failed to get container status \"cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411\": rpc error: code = NotFound desc = could not find container \"cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411\": container with ID starting with cf6fbde049d0e795c000b8af0ac32353e8aa07e7dcef3dd16b5ee38086bab411 not found: ID does not exist"
Nov 03 17:51:11.564279 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.564245    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkgk\" (UniqueName: \"kubernetes.io/projected/3f4f6054-05f5-41c5-848f-b245cbff25d3-kube-api-access-8lkgk\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.877286 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.877249    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"]
Nov 03 17:51:11.883806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:11.883774    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jv82l"]
Nov 03 17:51:12.117292 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:12.117256    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" path="/var/lib/kubelet/pods/3f4f6054-05f5-41c5-848f-b245cbff25d3/volumes"
Nov 03 17:51:15.282603 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:15.282563    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:20.389000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:20.388961    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:25.426119 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:25.426078    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:30.602206 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:30.602158    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:31.723749 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:31.723706    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:51:35.693108 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:35.693067    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:36.203965 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:36.203932    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:40.341828 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:51:40.341781    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:51:41.013092 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:41.013052    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:46.092839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:46.092786    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:50.849971 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.849932    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"]
Nov 03 17:51:50.851435 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.851407    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy"
Nov 03 17:51:50.851435 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.851434    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy"
Nov 03 17:51:50.851633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.851453    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy"
Nov 03 17:51:50.851633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.851462    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy"
Nov 03 17:51:50.851633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.851587    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy"
Nov 03 17:51:50.851633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.851609    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4f6054-05f5-41c5-848f-b245cbff25d3" containerName="istio-proxy"
Nov 03 17:51:50.855807 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.855787    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.910976 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.910943    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7d463-openshift-default-dockercfg-wqnxq\""
Nov 03 17:51:50.930203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930151    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930403 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930229    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930403 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930262    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930494 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930436    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930533 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930498    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930594 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930569    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930884 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930860    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.930978 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.930927    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.931041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.931020    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:50.962946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:50.962906    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"]
Nov 03 17:51:51.031924 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.031876    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.031924 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.031927    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.031945    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.031996    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032023    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032068    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032101    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032148    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032187    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032552 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032479    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032552 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032517    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032642 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032619    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032660    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.032911 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.032891    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.034698 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.034673    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.034962 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.034938    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.095953 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.095913    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.096119 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.096014    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.173397 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.173365    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:51.203230 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.203196    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:51.661546 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.661514    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"]
Nov 03 17:51:51.662760 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:51:51.662692    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c797500_276e_4849_a923_c6fc153bafce.slice/crio-c86e93cf48f99a78c2ccb510f80cfbaa09d346756628cab511359955f4627f1f WatchSource:0}: Error finding container c86e93cf48f99a78c2ccb510f80cfbaa09d346756628cab511359955f4627f1f: Status 404 returned error can't find the container with id c86e93cf48f99a78c2ccb510f80cfbaa09d346756628cab511359955f4627f1f
Nov 03 17:51:51.665246 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.665214    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:51:51.665319 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.665288    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:51:51.665356 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.665318    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:51:51.737632 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:51.737600    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" event={"ID":"6c797500-276e-4849-a923-c6fc153bafce","Type":"ContainerStarted","Data":"c86e93cf48f99a78c2ccb510f80cfbaa09d346756628cab511359955f4627f1f"}
Nov 03 17:51:52.745978 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:52.745941    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" event={"ID":"6c797500-276e-4849-a923-c6fc153bafce","Type":"ContainerStarted","Data":"72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e"}
Nov 03 17:51:52.924691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:52.924627    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podStartSLOduration=2.9246107070000003 podStartE2EDuration="2.924610707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:52.92125952 +0000 UTC m=+1048.931997629" watchObservedRunningTime="2026-11-03 17:51:52.924610707 +0000 UTC m=+1048.935348757"
Nov 03 17:51:53.173632 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:53.173588    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:53.175575 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:53.175546    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 17:51:53.175746 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:53.175606    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 17:51:54.174213 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:54.174181    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 17:51:54.174644 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:54.174235    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 17:51:55.174309 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:55.174271    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 17:51:55.174760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:55.174329    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 17:51:56.173997 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:56.173957    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 17:51:56.174209 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:56.174017    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 17:51:56.372897 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:56.372855    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:57.174192 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:57.174152    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 17:51:57.174410 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:57.174216    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 17:51:58.173989 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:58.173946    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 17:51:58.174410 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:58.174017    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 17:51:59.179066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:59.179029    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:59.179563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:59.179440    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:51:59.180163 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:51:59.180144    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 17:52:01.866797 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:01.866765    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:07.001073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:07.001040    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:09.943033 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:09.942990    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:10.338854 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:52:10.338718    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:52:12.311405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:12.311373    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:17.531996 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:17.531952    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:18.566862 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:18.566822    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:52:22.773069 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:22.773026    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:27.820458 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:27.820418    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:33.021570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:33.021475    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:34.945241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:34.945204    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:36.551195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:36.551155    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:38.225128 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:38.225088    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:40.340540 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:52:40.340495    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:52:43.313274 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:43.313243    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:48.409916 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:48.409882    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:53.515949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:53.515870    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:58.610610 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:52:58.610558    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:01.891652 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:01.891613    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"]
Nov 03 17:53:01.897319 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:01.897293    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.001849 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.001776    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:02.002300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.002270    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"distinct-w0-07f297-kserve-self-signed-certs\""
Nov 03 17:53:02.009087 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.009035    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"]
Nov 03 17:53:02.066530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.066491    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.066715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.066662    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.066715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.066699    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.066823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.066724    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhlz\" (UniqueName: \"kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.066859 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.066824    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.066898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.066886    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168357 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168257    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168357 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168303    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168606 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168436    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhlz\" (UniqueName: \"kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168606 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168493    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168606 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168594    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168782 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168648    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168782 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168675    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168877    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.168995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.168976    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.171417 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.171368    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.171556 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.171418    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.232709 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.232672    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhlz\" (UniqueName: \"kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz\") pod \"distinct-w0-07f297-kserve-5cb5886948-59h2r\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.510664 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.510566    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:02.895046 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:53:02.895010    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod706a5feb_cddf_4190_89a3_75f44cd6e13d.slice/crio-d0ba7ed1df4b99a0dc83194a03873826aed65411e80f0119f45d5151db533651 WatchSource:0}: Error finding container d0ba7ed1df4b99a0dc83194a03873826aed65411e80f0119f45d5151db533651: Status 404 returned error can't find the container with id d0ba7ed1df4b99a0dc83194a03873826aed65411e80f0119f45d5151db533651
Nov 03 17:53:02.961966 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:02.961937    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"]
Nov 03 17:53:03.253367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:03.253265    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" event={"ID":"706a5feb-cddf-4190-89a3-75f44cd6e13d","Type":"ContainerStarted","Data":"3801393122e782ff6d633a1da76a3064150b9f1affac458719b918d3f091204b"}
Nov 03 17:53:03.253367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:03.253313    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" event={"ID":"706a5feb-cddf-4190-89a3-75f44cd6e13d","Type":"ContainerStarted","Data":"d0ba7ed1df4b99a0dc83194a03873826aed65411e80f0119f45d5151db533651"}
Nov 03 17:53:03.374689 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:03.374627    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podStartSLOduration=2.374611232 podStartE2EDuration="2.374611232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:03.373977194 +0000 UTC m=+1119.384715265" watchObservedRunningTime="2026-11-03 17:53:03.374611232 +0000 UTC m=+1119.385349303"
Nov 03 17:53:03.510751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:03.510650    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:03.719358 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:03.719325    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:05.987301 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:05.987257    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"]
Nov 03 17:53:05.991506 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:05.991483    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.016122 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.016082    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttlf\" (UniqueName: \"kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.016317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.016146    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.016317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.016250    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.016317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.016281    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.016460 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.016429    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.016498 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.016462    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.084425 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.084389    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"]
Nov 03 17:53:06.098006 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.097966    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"openshift-service-ca.crt\""
Nov 03 17:53:06.098006 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.097968    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"kube-root-ca.crt\""
Nov 03 17:53:06.098273 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.098104    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"distinct-w6-f7d463-kserve-self-signed-certs\""
Nov 03 17:53:06.098412 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.098398    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"default-dockercfg-snqpt\""
Nov 03 17:53:06.117615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.117583    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.117842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.117640    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.117842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.117782    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttlf\" (UniqueName: \"kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.117970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.117893    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.117970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.117929    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.117970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.117953    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.118422 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.118391    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.118555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.118403    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.118555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.118483    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.120224 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.120192    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:06.120658 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.120636    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.121142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.121117    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.181804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.181777    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttlf\" (UniqueName: \"kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf\") pod \"distinct-w6-f7d463-kserve-775f978ff5-fz8zj\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.305081 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.304974    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:06.691176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:06.691145    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"]
Nov 03 17:53:06.692295 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:53:06.692270    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode84eff86_2993_4d78_a738_7d332dfbd042.slice/crio-f619654f55fe0445d2b4ca22158d507e0e624d46b9f73569b97fc80090bc6172 WatchSource:0}: Error finding container f619654f55fe0445d2b4ca22158d507e0e624d46b9f73569b97fc80090bc6172: Status 404 returned error can't find the container with id f619654f55fe0445d2b4ca22158d507e0e624d46b9f73569b97fc80090bc6172
Nov 03 17:53:07.281445 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:07.281402    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" event={"ID":"e84eff86-2993-4d78-a738-7d332dfbd042","Type":"ContainerStarted","Data":"ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1"}
Nov 03 17:53:07.281445 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:07.281449    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" event={"ID":"e84eff86-2993-4d78-a738-7d332dfbd042","Type":"ContainerStarted","Data":"f619654f55fe0445d2b4ca22158d507e0e624d46b9f73569b97fc80090bc6172"}
Nov 03 17:53:07.305396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:07.305356    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:07.406892 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:07.406834    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podStartSLOduration=2.406817533 podStartE2EDuration="2.406817533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:07.403945215 +0000 UTC m=+1123.414683289" watchObservedRunningTime="2026-11-03 17:53:07.406817533 +0000 UTC m=+1123.417555605"
Nov 03 17:53:08.803436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:08.803380    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:10.341878 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:53:10.341838    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:53:12.511107 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:12.511066    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:12.524164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:12.524132    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:13.350614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:13.350576    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:53:13.623848 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:13.623672    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:53:13.798930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:13.798880    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:14.200562 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:14.200527    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:16.305508 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:16.305463    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:16.318359 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:16.318333    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:16.383180 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:16.383149    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:53:18.918023 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:18.917987    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:22.127050 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:22.127018    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:23.995104 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:23.995073    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:27.685868 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.685830    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"]
Nov 03 17:53:27.690060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.690039    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.776605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776573    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.776857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776667    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:27.776857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776684    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.776857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776718    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.776857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776780    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.777031 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776860    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.777031 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.776896    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzcq\" (UniqueName: \"kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.794539 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.794510    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"]
Nov 03 17:53:27.796946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.796924    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"openshift-service-ca.crt\""
Nov 03 17:53:27.797028 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.797011    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"kube-root-ca.crt\""
Nov 03 17:53:27.797063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.797051    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"default-dockercfg-cqnvp\""
Nov 03 17:53:27.797601 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.797585    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"distinct-w5-548ca1-kserve-self-signed-certs\""
Nov 03 17:53:27.878060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878029    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878111    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878130    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878158    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878192    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878233    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzcq\" (UniqueName: \"kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878446    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878519    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.878651 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.878594    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.881842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.881799    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.882018 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.881999    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:27.943954 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:27.943860    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzcq\" (UniqueName: \"kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq\") pod \"distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:28.002876 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:28.002837    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:28.411428 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:28.411398    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"]
Nov 03 17:53:28.413127 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:53:28.413091    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee1d2385_7689_4ff5_8f33_8c043448c25c.slice/crio-41cf8e3b16e98abf8b32069706c2bc58ca360c92d9aa9405cd531608b49d468a WatchSource:0}: Error finding container 41cf8e3b16e98abf8b32069706c2bc58ca360c92d9aa9405cd531608b49d468a: Status 404 returned error can't find the container with id 41cf8e3b16e98abf8b32069706c2bc58ca360c92d9aa9405cd531608b49d468a
Nov 03 17:53:28.465011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:28.464978    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" event={"ID":"ee1d2385-7689-4ff5-8f33-8c043448c25c","Type":"ContainerStarted","Data":"41cf8e3b16e98abf8b32069706c2bc58ca360c92d9aa9405cd531608b49d468a"}
Nov 03 17:53:29.098835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:29.098802    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:29.474659 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:29.474623    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" event={"ID":"ee1d2385-7689-4ff5-8f33-8c043448c25c","Type":"ContainerStarted","Data":"4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb"}
Nov 03 17:53:29.597889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:29.597831    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podStartSLOduration=2.597816564 podStartE2EDuration="2.597816564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:29.595145735 +0000 UTC m=+1145.605883830" watchObservedRunningTime="2026-11-03 17:53:29.597816564 +0000 UTC m=+1145.608554635"
Nov 03 17:53:30.003440 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:30.003399    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:34.200368 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:34.200334    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:36.895433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:36.895389    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:38.003964 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:38.003925    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:38.017112 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:38.017078    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:38.559753 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:38.559698    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:53:39.310755 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:39.310618    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:40.357433 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:53:40.357392    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:53:41.045430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:41.045395    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:44.519828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:44.519775    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:49.606158 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:49.606116    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:54.467519 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.467484    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"]
Nov 03 17:53:54.471535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.471509    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.580418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.580370    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"]
Nov 03 17:53:54.638614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.638573    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.638844 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.638621    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.638844 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.638798    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.638959 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.638844    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k88s\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.639021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.638983    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.639099 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.639079    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.639209 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.639192    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.639255 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.639238    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.639303 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.639274    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.691622 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.691580    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:54.740298 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740195    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740298 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740275    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740316    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740366    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740403    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740482    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740517    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k88s\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740710 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740570    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.740710 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.740631    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.741570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.741543    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.741872 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.741850    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.742483 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.742457    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.742789 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.742714    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.743067 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.742921    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.744346 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.744325    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.745217 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.745199    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.805950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.805916    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:54.806108 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:54.806019    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k88s\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:55.090938 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.090844    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:55.349568 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.349483    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:55.503614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.503559    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:55.530320 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.530275    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"]
Nov 03 17:53:55.563333 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:53:55.563306    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb222a6c8_acbb_42a0_8a33_e5056c50458d.slice/crio-22624fd3c654cfbe7b7a1a406056ccb3cf6fa64bd8a39d3ba2d50425b364edda WatchSource:0}: Error finding container 22624fd3c654cfbe7b7a1a406056ccb3cf6fa64bd8a39d3ba2d50425b364edda: Status 404 returned error can't find the container with id 22624fd3c654cfbe7b7a1a406056ccb3cf6fa64bd8a39d3ba2d50425b364edda
Nov 03 17:53:55.565882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.565846    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:55.565952 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.565921    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:55.565994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.565950    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:55.685677 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.681710    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" event={"ID":"b222a6c8-acbb-42a0-8a33-e5056c50458d","Type":"ContainerStarted","Data":"8eff44e4f7b968059ae42c75180296fd169f1d8cb2ab802d1450095ea3d2bc31"}
Nov 03 17:53:55.685677 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.681845    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" event={"ID":"b222a6c8-acbb-42a0-8a33-e5056c50458d","Type":"ContainerStarted","Data":"22624fd3c654cfbe7b7a1a406056ccb3cf6fa64bd8a39d3ba2d50425b364edda"}
Nov 03 17:53:55.807590 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.807537    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podStartSLOduration=1.80751763 podStartE2EDuration="1.80751763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:55.805491833 +0000 UTC m=+1171.816229908" watchObservedRunningTime="2026-11-03 17:53:55.80751763 +0000 UTC m=+1171.818255703"
Nov 03 17:53:55.975652 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:55.975600    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:56.091113 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:56.091075    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:53:56.092629 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:56.092596    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:56.092783 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:56.092660    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:53:56.145685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:56.145650    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:57.091929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:57.091890    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:57.092359 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:57.091956    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:53:58.091461 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:58.091426    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:58.091650 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:58.091488    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:53:59.092268 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:59.092229    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:59.092696 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:59.092292    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:53:59.729037 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:53:59.729001    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:00.091948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:00.091861    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:54:00.091948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:00.091917    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:54:01.091599 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:01.091562    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:54:01.092025 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:01.091623    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:54:02.091393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:02.091357    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:54:02.091674 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:02.091414    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 17:54:03.096586 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:03.096506    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:54:03.097037 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:03.096908    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:54:03.097644 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:03.097623    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 17:54:04.708805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:04.708767    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:54:04.884517 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:04.884481    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:07.104034 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.104003    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:07.329004 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.328960    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"]
Nov 03 17:54:07.334366 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.334335    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425143 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425095    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425159    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425186    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425232    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425279    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv442\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425513    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425635    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425660    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.425708 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.425684    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.446885 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.446845    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"]
Nov 03 17:54:07.527418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527375    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv442\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527484    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527626    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527658    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527699    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527778    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527914 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527870    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.527914 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527901    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.528012 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.527959    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.528432 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.528357    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.528560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.528429    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.528560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.528486    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.528560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.528502    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.528794 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.528675    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.530238 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.530211    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.530359 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.530344    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.591101 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.591071    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.591279 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.591209    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv442\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:07.651413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:07.651374    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:08.089268 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.089238    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"]
Nov 03 17:54:08.090517 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:54:08.090488    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeba4954_d3f8_4cd5_8ffe_a58bbf3eda50.slice/crio-39256c7edcdf5554e024f9ee2e4fbbf5534ca82829d9dc64ca7ce763dc94ec84 WatchSource:0}: Error finding container 39256c7edcdf5554e024f9ee2e4fbbf5534ca82829d9dc64ca7ce763dc94ec84: Status 404 returned error can't find the container with id 39256c7edcdf5554e024f9ee2e4fbbf5534ca82829d9dc64ca7ce763dc94ec84
Nov 03 17:54:08.093205 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.093166    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:54:08.093321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.093240    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:54:08.093321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.093289    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:54:08.830421 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.830381    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" event={"ID":"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50","Type":"ContainerStarted","Data":"d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf"}
Nov 03 17:54:08.830878 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.830444    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" event={"ID":"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50","Type":"ContainerStarted","Data":"39256c7edcdf5554e024f9ee2e4fbbf5534ca82829d9dc64ca7ce763dc94ec84"}
Nov 03 17:54:08.958593 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:08.958538    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podStartSLOduration=1.958522575 podStartE2EDuration="1.958522575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:08.956873801 +0000 UTC m=+1184.967611874" watchObservedRunningTime="2026-11-03 17:54:08.958522575 +0000 UTC m=+1184.969260646"
Nov 03 17:54:09.652206 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:09.652168    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:09.654273 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:09.654241    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:09.654392 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:09.654301    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 17:54:10.134287 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:10.134179    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:10.356005 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:54:10.355964    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:54:10.651868 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:10.651827    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:10.652051 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:10.651898    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 17:54:11.652570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:11.652532    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:11.653020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:11.652623    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 17:54:12.652406 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:12.652358    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:12.652600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:12.652430    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 17:54:13.652416 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:13.652369    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:13.652647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:13.652462    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 17:54:14.651852 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:14.651813    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:14.652063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:14.651880    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 17:54:15.296054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:15.296014    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:15.657153 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:15.657122    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:15.657593 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:15.657578    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:15.658437 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:15.658417    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 17:54:20.417250 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:20.417208    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:24.324049 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:24.324018    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:54:24.331398 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:24.331375    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:54:25.419080 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:25.419044    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:30.473606 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:30.473573    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:35.714925 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:35.714890    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:35.744250 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:35.744212    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:54:37.263341 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:37.263302    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:37.487891 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:37.487856    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:40.341535 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:54:40.341494    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:54:40.736902 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:40.736865    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:45.846790 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:45.846750    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:46.220524 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:46.220489    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:48.192840 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:48.192805    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:51.009014 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:51.008981    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:56.245375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:54:56.245321    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:01.405846 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:01.405800    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:06.141136 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:06.141100    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:55:06.502892 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:06.502786    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:10.340358 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:55:10.340309    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:55:11.601539 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:11.601466    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:16.707654 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:16.707592    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:21.305998 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.305958    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"]
Nov 03 17:55:21.310333 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.310308    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.394492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.394454    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.394690 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.394522    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.394690 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.394643    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.394812 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.394796    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnt6\" (UniqueName: \"kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.394856 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.394842    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.394897 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.394885    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.414256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.414223    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"distinct-2-w6-f7d463-kserve-self-signed-certs\""
Nov 03 17:55:21.417471 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.417445    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"]
Nov 03 17:55:21.425643 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.425617    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:21.495833 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.495793    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnt6\" (UniqueName: \"kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.496035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.495864    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.496035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.495893    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.496035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.495920    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.496035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.495975    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.496035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.496023    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.496468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.496449    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.497467 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.497441    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.497659 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.497549    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.499235 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.499213    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.500302 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.500277    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.563014 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.562938    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnt6\" (UniqueName: \"kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6\") pod \"distinct-2-w6-f7d463-kserve-869b4754b8-znvtn\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") " pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.625091 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.625035    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:21.819713 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:21.819623    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:22.056457 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:22.056428    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"]
Nov 03 17:55:22.057401 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:55:22.057371    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0b18255_4496_4e01_a9b5_2106b4493edd.slice/crio-8d714cff0ce08637843f214c5610e50e0f22e20ef14df1c2cab054a81bf8944e WatchSource:0}: Error finding container 8d714cff0ce08637843f214c5610e50e0f22e20ef14df1c2cab054a81bf8944e: Status 404 returned error can't find the container with id 8d714cff0ce08637843f214c5610e50e0f22e20ef14df1c2cab054a81bf8944e
Nov 03 17:55:22.390939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:22.390906    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" event={"ID":"d0b18255-4496-4e01-a9b5-2106b4493edd","Type":"ContainerStarted","Data":"9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354"}
Nov 03 17:55:22.391335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:22.390949    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" event={"ID":"d0b18255-4496-4e01-a9b5-2106b4493edd","Type":"ContainerStarted","Data":"8d714cff0ce08637843f214c5610e50e0f22e20ef14df1c2cab054a81bf8944e"}
Nov 03 17:55:22.512751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:22.512681    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podStartSLOduration=1.5126668479999998 podStartE2EDuration="1.512666848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:22.510628808 +0000 UTC m=+1258.521366906" watchObservedRunningTime="2026-11-03 17:55:22.512666848 +0000 UTC m=+1258.523404920"
Nov 03 17:55:22.625334 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:22.625146    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:26.967269 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:26.967208    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:28.211843 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:28.211809    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:31.625353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:31.625305    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:31.637999 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:31.637970    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:32.012835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:32.012798    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:32.481985 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:32.481898    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:55:37.209066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.209025    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:37.631998 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.631918    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:37.656897 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.656862    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"]
Nov 03 17:55:37.661233 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.661212    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693290 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693246    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693488 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693386    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693488 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693417    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693488 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693476    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693629 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693573    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6zm\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693690 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693649    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693783 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693712    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693864 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693845    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.693984 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.693969    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.774061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.774022    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"]
Nov 03 17:55:37.795565 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795521    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.795785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795603    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.795785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795678    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.795785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795707    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.795785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795750    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.796009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795787    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6zm\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.796009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795889    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.796009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795925    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.796009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.795990    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.796220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.796103    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.796573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.796505    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.797400    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.797477    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.797725    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.799061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.799035    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.800266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.800241    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.865468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.865438    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.865633 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.865487    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6zm\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:37.975464 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:37.975425    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:38.449174 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.449144    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"]
Nov 03 17:55:38.449802 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:55:38.449773    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6489f_0097_4ff1_9658_3e8a90edf1c6.slice/crio-10fdbbe665d8144a2350a3b44028844a95d0f898be90843fc679906afbc6da33 WatchSource:0}: Error finding container 10fdbbe665d8144a2350a3b44028844a95d0f898be90843fc679906afbc6da33: Status 404 returned error can't find the container with id 10fdbbe665d8144a2350a3b44028844a95d0f898be90843fc679906afbc6da33
Nov 03 17:55:38.452370 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.452336    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:55:38.452442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.452413    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:55:38.452482 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.452444    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:55:38.526683 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.526639    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" event={"ID":"41f6489f-0097-4ff1-9658-3e8a90edf1c6","Type":"ContainerStarted","Data":"1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2"}
Nov 03 17:55:38.526843 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.526697    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" event={"ID":"41f6489f-0097-4ff1-9658-3e8a90edf1c6","Type":"ContainerStarted","Data":"10fdbbe665d8144a2350a3b44028844a95d0f898be90843fc679906afbc6da33"}
Nov 03 17:55:38.661665 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.661601    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podStartSLOduration=1.661585946 podStartE2EDuration="1.661585946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:38.659089626 +0000 UTC m=+1274.669827700" watchObservedRunningTime="2026-11-03 17:55:38.661585946 +0000 UTC m=+1274.672324017"
Nov 03 17:55:38.976181 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:38.976138    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:39.978017 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:39.977981    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:55:39.978422 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:39.978040    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 17:55:40.360249 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:55:40.360154    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:55:40.976714 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:40.976672    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:55:40.976970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:40.976777    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 17:55:41.336224 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:41.336130    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:55:41.976258 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:41.976222    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:55:41.976457 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:41.976281    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 17:55:42.596855 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:42.596802    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:42.976182 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:42.976146    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:55:42.976485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:42.976452    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 17:55:43.978318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:43.978277    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:55:43.979054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:43.979021    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 17:55:44.976718 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:44.976623    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:55:44.976718 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:44.976688    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 17:55:46.162699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:46.162667    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:46.163234 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:46.162977    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:46.164015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:46.163994    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 17:55:47.910781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:47.910749    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:53.010252 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:53.010206    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:53.914933 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:53.914896    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:58.155324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:55:58.155203    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:03.190572 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:03.190531    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:05.331289 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:05.331245    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:08.012078 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.012044    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"]
Nov 03 17:56:08.016699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.016666    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.052557 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052520    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.052799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052636    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.052799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052678    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.052799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052712    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.052799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052791    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.053058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052818    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.053058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052881    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.053058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052910    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.053058 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.052936    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhxv\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.128580 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.128538    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"]
Nov 03 17:56:08.154517 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154473    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.154758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154534    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.154758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154578    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.154758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154616    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.154758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154658    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhxv\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154781    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154876    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154916    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154965    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155015 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.154981    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.155529    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.155992 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.155668    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.156317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.155785    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.156449 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.155938    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.157717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.157677    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.158175 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.158151    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.219048 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.219006    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.219256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.219229    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhxv\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.331721 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.331619    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:08.412042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.412001    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:08.770834 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.770802    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"]
Nov 03 17:56:08.771462 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:56:08.771431    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd5658a_c97e_44b2_9375_43983eb9eac1.slice/crio-e91a70915177e91e6f96584e8fc1911f611ee55a1e68fcbee6d4dde560b68f50 WatchSource:0}: Error finding container e91a70915177e91e6f96584e8fc1911f611ee55a1e68fcbee6d4dde560b68f50: Status 404 returned error can't find the container with id e91a70915177e91e6f96584e8fc1911f611ee55a1e68fcbee6d4dde560b68f50
Nov 03 17:56:08.773981 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.773945    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:08.774094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.774024    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:08.774094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.774071    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:08.811196 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:08.811155    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" event={"ID":"6cd5658a-c97e-44b2-9375-43983eb9eac1","Type":"ContainerStarted","Data":"e91a70915177e91e6f96584e8fc1911f611ee55a1e68fcbee6d4dde560b68f50"}
Nov 03 17:56:09.861757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:09.855604    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" event={"ID":"6cd5658a-c97e-44b2-9375-43983eb9eac1","Type":"ContainerStarted","Data":"06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c"}
Nov 03 17:56:10.045219 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:10.045150    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podStartSLOduration=3.045128798 podStartE2EDuration="3.045128798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:10.040704819 +0000 UTC m=+1306.051442910" watchObservedRunningTime="2026-11-03 17:56:10.045128798 +0000 UTC m=+1306.055866868"
Nov 03 17:56:10.332274 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:10.332169    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:10.340220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:10.338858    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 17:56:10.340220 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:10.338925    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 17:56:10.376721 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:56:10.376677    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:56:11.333619 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.333583    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 17:56:11.334717 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.334685    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 17:56:11.451283 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.451239    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"]
Nov 03 17:56:11.458061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.458024    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.522874 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.522832    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"]
Nov 03 17:56:11.523444 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.523418    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"openshift-service-ca.crt\""
Nov 03 17:56:11.542759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.542700    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.543112 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.543088    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.543278 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.543257    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.543418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.543401    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.543547 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.543529    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzf7v\" (UniqueName: \"kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.543701 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.543685    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.547535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.547500    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:11.559715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.558401    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"kube-root-ca.crt\""
Nov 03 17:56:11.559715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.558529    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"default-dockercfg-czx7h\""
Nov 03 17:56:11.573715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.573685    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"distinct-2-w4-73a606-kserve-self-signed-certs\""
Nov 03 17:56:11.656503 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.656464    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.656720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.656682    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.656839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.656787    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.656839 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.656816    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.656959 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.656847    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.656959 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.656874    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzf7v\" (UniqueName: \"kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.657850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.657821    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.658159 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.658135    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.658760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.658564    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.661945 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.661920    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.662301 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.662080    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.729628 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.729597    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzf7v\" (UniqueName: \"kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v\") pod \"distinct-2-w4-73a606-kserve-856f677f69-d8t46\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:11.778753 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:11.778708    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:12.258974 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:12.258939    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"]
Nov 03 17:56:12.259581 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:56:12.259555    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98b8fa95_a4fa_4e85_a88e_ac177e5cb602.slice/crio-74a71553ef253daf65589b54fedf72ef94aad90fb72eb890ecabc4f69a04bce0 WatchSource:0}: Error finding container 74a71553ef253daf65589b54fedf72ef94aad90fb72eb890ecabc4f69a04bce0: Status 404 returned error can't find the container with id 74a71553ef253daf65589b54fedf72ef94aad90fb72eb890ecabc4f69a04bce0
Nov 03 17:56:12.331979 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:12.331935    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 17:56:12.332145 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:12.331998    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 17:56:12.923806 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:12.923766    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" event={"ID":"98b8fa95-a4fa-4e85-a88e-ac177e5cb602","Type":"ContainerStarted","Data":"765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26"}
Nov 03 17:56:12.924424 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:12.924393    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" event={"ID":"98b8fa95-a4fa-4e85-a88e-ac177e5cb602","Type":"ContainerStarted","Data":"74a71553ef253daf65589b54fedf72ef94aad90fb72eb890ecabc4f69a04bce0"}
Nov 03 17:56:13.047495 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:13.047434    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podStartSLOduration=2.04741742 podStartE2EDuration="2.04741742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:13.04381034 +0000 UTC m=+1309.054548413" watchObservedRunningTime="2026-11-03 17:56:13.04741742 +0000 UTC m=+1309.058155492"
Nov 03 17:56:13.332397 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:13.332302    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 17:56:13.332397 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:13.332368    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 17:56:13.688112 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:13.688038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:13.779362 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:13.779326    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:14.332703 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:14.332666    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 17:56:14.333168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:14.332726    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 17:56:15.332549 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:15.332505    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 17:56:15.332758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:15.332579    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 17:56:16.337062 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:16.337028    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:16.337563 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:16.337317    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:16.338205 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:16.338185    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 17:56:18.929074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:18.929037    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:21.779566 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:21.779520    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:21.792041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:21.792017    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:22.007667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:22.007636    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:56:23.797477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.797435    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"]
Nov 03 17:56:23.802215 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.802182    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847334 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847297    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847533 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847362    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx72\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847533 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847494    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847533 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847524    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847542    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847671 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847650    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847747 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847703    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847747 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847719    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.847908 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.847882    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.911793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.911750    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"]
Nov 03 17:56:23.948866 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.948830    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx72\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.948919    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.948944    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.948964    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.948983    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949071 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949050    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949074    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949120    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949469 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949234    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949663 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949541    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949663 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949553    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.949902 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949867    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.950046 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.949938    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.950046 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.950010    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.952155 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.952124    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:23.955250 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:23.955213    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:24.001595 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.001558    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"]
Nov 03 17:56:24.007261 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.007228    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.012458 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.012422    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:24.013456 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.013433    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx72\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:24.050289 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050211    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050289 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050264    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050420    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050530 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050510    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdk6r\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050602 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050588    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050644 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050630    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050679 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050669    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050714    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.050869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.050853    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.069885 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.069850    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"]
Nov 03 17:56:24.102280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.102241    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:24.123139 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.123110    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157310    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdk6r\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157421    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157478    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157547    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157600    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157863    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157940    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.157980    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.158108    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.159886    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.160212    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.160687    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.161450    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.162049    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.162830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.162356    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.165439 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.165413    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.225599 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.225069    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdk6r\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.235574 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.235111    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.347955 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.347864    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:24.597022 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.596803    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"]
Nov 03 17:56:24.597570 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:56:24.597541    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2d2de9_bde5_4783_9ffb_afb416501fd6.slice/crio-fb2946c7a2cbb555a63d02445a70875eceb51a0794483614cbee8bbae3403b72 WatchSource:0}: Error finding container fb2946c7a2cbb555a63d02445a70875eceb51a0794483614cbee8bbae3403b72: Status 404 returned error can't find the container with id fb2946c7a2cbb555a63d02445a70875eceb51a0794483614cbee8bbae3403b72
Nov 03 17:56:24.600686 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.600649    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.600830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.600723    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.600830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.600770    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.748083 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.748051    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"]
Nov 03 17:56:24.751904 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:56:24.751873    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4fc713_c468_424a_85bf_130d5552ec71.slice/crio-3589b7461cf52c962204cb372ed9a10a37560ad83d4be7eebf1b4612a4f81339 WatchSource:0}: Error finding container 3589b7461cf52c962204cb372ed9a10a37560ad83d4be7eebf1b4612a4f81339: Status 404 returned error can't find the container with id 3589b7461cf52c962204cb372ed9a10a37560ad83d4be7eebf1b4612a4f81339
Nov 03 17:56:24.754721 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.754613    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.754944 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.754918    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.755082 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:24.755065    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:25.015803 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.015708    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" event={"ID":"5c2d2de9-bde5-4783-9ffb-afb416501fd6","Type":"ContainerStarted","Data":"c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25"}
Nov 03 17:56:25.016912 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.015821    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" event={"ID":"5c2d2de9-bde5-4783-9ffb-afb416501fd6","Type":"ContainerStarted","Data":"fb2946c7a2cbb555a63d02445a70875eceb51a0794483614cbee8bbae3403b72"}
Nov 03 17:56:25.018097 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.018068    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" event={"ID":"6c4fc713-c468-424a-85bf-130d5552ec71","Type":"ContainerStarted","Data":"679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be"}
Nov 03 17:56:25.018241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.018126    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" event={"ID":"6c4fc713-c468-424a-85bf-130d5552ec71","Type":"ContainerStarted","Data":"3589b7461cf52c962204cb372ed9a10a37560ad83d4be7eebf1b4612a4f81339"}
Nov 03 17:56:25.124353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.124313    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:25.126443 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.126412    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:25.126605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.126491    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:25.140627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.140564    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podStartSLOduration=2.140544942 podStartE2EDuration="2.140544942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:25.140500598 +0000 UTC m=+1321.151238671" watchObservedRunningTime="2026-11-03 17:56:25.140544942 +0000 UTC m=+1321.151283014"
Nov 03 17:56:25.267160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.267050    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podStartSLOduration=2.267035933 podStartE2EDuration="2.267035933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:25.264674243 +0000 UTC m=+1321.275412390" watchObservedRunningTime="2026-11-03 17:56:25.267035933 +0000 UTC m=+1321.277774047"
Nov 03 17:56:25.348074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.348038    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:25.350384 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.350351    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:25.350573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:25.350410    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:26.127366 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:26.127327    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:26.128082 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:26.128044    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:26.352342 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:26.352302    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:26.352689 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:26.352643    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:27.136757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:27.136440    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:27.136757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:27.136499    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:27.351318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:27.351279    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:27.351616 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:27.351589    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:28.163778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:28.163362    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:28.163778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:28.163451    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:28.351754 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:28.351206    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:28.351754 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:28.351314    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:29.126759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:29.126193    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:29.126759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:29.126257    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:29.360759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:29.355263    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:29.360759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:29.355341    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:29.377823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:29.371907    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:29.448841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:29.448799    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:56:30.150759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:30.150140    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:30.150759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:30.150216    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:30.363756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:30.358111    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:30.363756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:30.358174    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:31.152758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:31.141936    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:31.152758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:31.142003    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:31.364756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:31.353371    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:31.364756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:31.353438    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:32.128757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:32.128634    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:56:32.128757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:32.128710    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 17:56:32.349438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:32.348945    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 17:56:32.349438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:32.349011    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 17:56:33.354938 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:33.354896    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:33.355488 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:33.355316    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:33.356040 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:33.356017    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 17:56:34.165922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:34.165883    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:56:34.166135 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:34.165954    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:56:34.273916 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:34.273883    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:34.274355 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:34.274332    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:34.275124 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:34.275106    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 17:56:34.719100 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:34.719067    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:37.951341 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:37.951301    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:39.823571 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:39.823518    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:40.340877 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:56:40.340836    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:56:45.037841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:45.037704    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:50.117202 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:50.117145    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:54.354777 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.350251    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:56:54.483764 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.482861    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:56:54.601801 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.560642    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:56:54.702796 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.700324    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"]
Nov 03 17:56:54.781759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.761120    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"]
Nov 03 17:56:54.781759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.761294    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:54.908769 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.885447    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-73a606-openshift-default-dockercfg-6xzts\""
Nov 03 17:56:54.943353 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.912350    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:54.943640 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.943616    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:54.943785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.943766    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:54.943926 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.943912    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:54.944018 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.944005    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:54.970829 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.944104    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chg6\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.014770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.014719    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.038761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.015199    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.039156 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.039120    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.039473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:54.994816    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:56:55.146102 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.146060    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.171057 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.171002    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.266880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.171345    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.267172 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.267140    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.267358 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.267333    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.267486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.267468    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.267607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.267587    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chg6\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.268099 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.268070    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.268240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.208107    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:56:55.269673 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.159466    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.272525 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.272498    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:55.292393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.292338    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.298648 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.298613    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.310759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.299166    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.311855 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.311820    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.318403 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.318369    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.333757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.333281    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.352764 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.352699    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:56:55.354498 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.354470    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:56:55.377795 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.373669    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.377795 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.376325    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:56:55.503770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.500961    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:56:55.503770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.501536    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chg6\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.507766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.506450    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:56:55.514957 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.514891    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.661979 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.661946    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:55.782984 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.782488    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Nov 03 17:56:55.791615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.791026    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:56:55.796298 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:55.796271    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:56:56.211195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.211057    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"]
Nov 03 17:56:56.235842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.235802    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.250307 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.238139    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:56.250307 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.243750    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"]
Nov 03 17:56:56.420308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420264    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chp8d\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.420830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420319    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.420830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420454    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.420830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420490    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.420830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420506    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.420830 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420712    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.421042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420831    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.421042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420857    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.421042 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.420875    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522597 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522498    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522597 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522552    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522597 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522577    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522776    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chp8d\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522814    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522887    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522913    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.522950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.522937    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.523216 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.523078    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.523216 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.523075    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.524171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.523415    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.524171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.523440    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.524171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.523647    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.524171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.524071    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.525919 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.525893    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.526061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.526007    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.586598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.586556    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.586827 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.586808    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chp8d\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:56.608293 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.608260    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"]
Nov 03 17:56:56.612862 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:56:56.612831    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5415ca1c_de9f_41e4_90ff_7ad756afedb2.slice/crio-6a49f29a67f8d9720519775a457b6b442573f3d1d53836da01f7ac424b44c3d4 WatchSource:0}: Error finding container 6a49f29a67f8d9720519775a457b6b442573f3d1d53836da01f7ac424b44c3d4: Status 404 returned error can't find the container with id 6a49f29a67f8d9720519775a457b6b442573f3d1d53836da01f7ac424b44c3d4
Nov 03 17:56:56.615577 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.615540    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:56.615697 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.615635    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:56.615697 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.615682    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:56.879872 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:56.879773    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:57.052772 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.052719    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" event={"ID":"5415ca1c-de9f-41e4-90ff-7ad756afedb2","Type":"ContainerStarted","Data":"185e2bde2f9b4dac19c675f2cbfbf6d19107c374a0c8e9d90b5bdb8a5ce035d1"}
Nov 03 17:56:57.052969 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.052797    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" event={"ID":"5415ca1c-de9f-41e4-90ff-7ad756afedb2","Type":"ContainerStarted","Data":"6a49f29a67f8d9720519775a457b6b442573f3d1d53836da01f7ac424b44c3d4"}
Nov 03 17:56:57.183643 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.183586    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podStartSLOduration=3.183564005 podStartE2EDuration="3.183564005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:57.180112525 +0000 UTC m=+1353.190850620" watchObservedRunningTime="2026-11-03 17:56:57.183564005 +0000 UTC m=+1353.194302077"
Nov 03 17:56:57.360034 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.359920    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"]
Nov 03 17:56:57.373949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.373462    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:57.373949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.373536    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:57.373949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.373565    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:57.663256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.663221    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:56:57.664507 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.664481    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:57.664579 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:57.664542    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:56:58.062063 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.062023    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" event={"ID":"1c318f8a-017b-4ca7-874d-4e95cc50fb56","Type":"ContainerStarted","Data":"c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497"}
Nov 03 17:56:58.062276 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.062076    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" event={"ID":"1c318f8a-017b-4ca7-874d-4e95cc50fb56","Type":"ContainerStarted","Data":"561b8025f076930852578b43fb87e4c853b7af2483b6409c52d63fb4c338d413"}
Nov 03 17:56:58.191133 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.191076    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podStartSLOduration=3.191061278 podStartE2EDuration="3.191061278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:58.188541284 +0000 UTC m=+1354.199279370" watchObservedRunningTime="2026-11-03 17:56:58.191061278 +0000 UTC m=+1354.201799349"
Nov 03 17:56:58.662626 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.662586    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:58.662849 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.662638    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:56:58.880266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.880237    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:56:58.882086 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.882064    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:56:58.882184 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:58.882111    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:56:59.663091 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:59.663058    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:59.663344 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:59.663317    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:56:59.880946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:59.880903    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:56:59.881324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:56:59.880960    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:57:00.664186 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:00.662521    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:57:00.664186 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:00.662596    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:57:00.880442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:00.880405    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:57:00.880627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:00.880468    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:57:01.663338 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:01.663292    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:57:01.663916 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:01.663357    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:57:01.693455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:01.693423    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:01.881084 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:01.880597    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:57:01.881084 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:01.880665    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:57:02.241842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:02.241648    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:02.426793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:02.425419    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:02.663376 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:02.663336    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:57:02.663835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:02.663409    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:57:02.881162 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:02.881131    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:57:02.881400 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:02.881378    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:57:03.663108 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:03.663074    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:57:03.663327 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:03.663132    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 17:57:03.880745 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:03.880696    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:57:03.881180 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:03.880790    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:57:04.668289 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:04.668257    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:57:04.668759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:04.668722    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:57:04.669773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:04.669721    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 17:57:04.880667 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:04.880631    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 17:57:04.880882 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:04.880687    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 17:57:05.885019 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:05.884988    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:57:05.885460 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:05.885391    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:57:05.886175 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:05.886157    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 17:57:06.897518 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:06.897485    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:08.465502 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.465468    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"]
Nov 03 17:57:08.471078 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.471050    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494393 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494355    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494566 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494416    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494566 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494469    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494566 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494503    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494675 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494611    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494711    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7trj\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494813 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494799    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494845 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494827    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.494967 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.494954    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.568200 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.567598    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"]
Nov 03 17:57:08.574145 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.574119    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.583155 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.583116    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"]
Nov 03 17:57:08.598593 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.598562    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.598798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.598613    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.598798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.598662    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.598798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.598679    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.598798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.598699    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.598798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.598721    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599009    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599051    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599127    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599154    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599248    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7trj\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599296    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwsc\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599334    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599402    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.599875 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599426    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.599875 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599456    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.599996 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.599891    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.600287 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.600265    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.600352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.600338    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.601277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.601255    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.601543 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.601524    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.601773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.601718    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.601880 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.601831    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.602651 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.602625    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.603834 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.603809    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.631322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.631277    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"]
Nov 03 17:57:08.639097 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.639064    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.649549 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.647088    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"]
Nov 03 17:57:08.671805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.671770    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7trj\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.672132 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.672106    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.695544 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.695498    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"]
Nov 03 17:57:08.702653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.702621    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.702940 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.702913    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwsc\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.703130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703110    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.703258 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703245    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.703367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703352    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.703503 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703488    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.703654 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703640    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.703770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703755    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.703894 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703878    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.703992 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.703979    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ntl\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.704135 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.704116    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.704288 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.704272    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.704423 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.704408    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.704553 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.704535    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.704681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.704667    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.704947 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.704915    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.705241 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.705221    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.705413 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.705397    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.705619 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.705593    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.705725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.705288    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.705725 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.705603    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.706341 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.706313    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.706441 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.706385    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.708129 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.708098    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.708549 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.708530    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.769111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.769027    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.769320 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.769296    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwsc\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.786610 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.786564    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:08.807126 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807095    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807240 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807199    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807303 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807253    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807363 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807348    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807382    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807478 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807414    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807478 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807444    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807585 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807478    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ntl\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807585 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807567    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.807684 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807593    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.808156 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.807939    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.808156 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.808047    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.808156 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.808142    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.808367 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.808305    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.810711 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.810682    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.811143 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.811123    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.856867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.856825    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"]
Nov 03 17:57:08.877090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.877041    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.883428 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.883395    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.889982 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.889950    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"]
Nov 03 17:57:08.896068 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.896035    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:08.898611 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.898565    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ntl\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:08.906986 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.904060    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:08.909672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.909161    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.909672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.909207    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910507 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910480    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910529    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910656    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnr8s\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910691    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910714    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910758    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.910842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.910819    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:08.933517 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.930362    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"]
Nov 03 17:57:08.951079 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.950510    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"]
Nov 03 17:57:08.962624 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:08.962401    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012309    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012357    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012418    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012446    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012495    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6b6\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012568    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012641    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnr8s\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012692    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012725    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012783    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012816    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012873    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012909    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012942    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.012985    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.013024    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.013082    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.013934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.013110    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.015000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.014602    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.015000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.014971    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.015580 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.015458    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.015580 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.015520    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.015853 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.015808    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.024756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.024050    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.035932 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.035871    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.079598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.079552    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnr8s\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.079986 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.079967    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.114898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.114864    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.114928    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.114959    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115094 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115073    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115264 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115102    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115264 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115248    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115369 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115280    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115369 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115347    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6b6\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115472 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115419    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.115472 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115421    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.116637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.115806    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.116637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.116144    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.116637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.116565    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.116637 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.116591    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.118435 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.118411    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.118641 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.118621    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.179915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.179884    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.180111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.180017    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6b6\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.210780 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.210720    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:09.223590 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.223548    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:09.225085 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.225056    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"]
Nov 03 17:57:09.232372 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.232348    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.246487 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:57:09.246366    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee56aec_eddf_4726_8ab6_b3a71133cf69.slice/crio-294beee60fd30c85d9aeff98119c08f19519f4c4f7f938f25176fef0b3718c99 WatchSource:0}: Error finding container 294beee60fd30c85d9aeff98119c08f19519f4c4f7f938f25176fef0b3718c99: Status 404 returned error can't find the container with id 294beee60fd30c85d9aeff98119c08f19519f4c4f7f938f25176fef0b3718c99
Nov 03 17:57:09.247181 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.247150    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"]
Nov 03 17:57:09.250011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.249975    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.250104 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.250045    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.250104 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.250075    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.292694 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.292611    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"]
Nov 03 17:57:09.328304 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.328249    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.328445 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.328347    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjpfw\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.328445 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.328393    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.328560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.328473    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.328560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.328510    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.328664 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.328561    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.329141 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.329048    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.329428 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.329403    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.329486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.329475    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.379492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.379242    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"]
Nov 03 17:57:09.381913 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.381766    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.382040 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.381854    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.382115 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.382088    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.396473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.396444    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"]
Nov 03 17:57:09.403267 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:57:09.403184    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddace43b_4112_4594_92df_095167b2bb3f.slice/crio-8861b75c6b997a1ae4d2b93bcb9d8c8123969c9f794b726bee143c47d3cffda4 WatchSource:0}: Error finding container 8861b75c6b997a1ae4d2b93bcb9d8c8123969c9f794b726bee143c47d3cffda4: Status 404 returned error can't find the container with id 8861b75c6b997a1ae4d2b93bcb9d8c8123969c9f794b726bee143c47d3cffda4
Nov 03 17:57:09.406570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.406440    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.406570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.406529    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.406951 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.406799    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.430940 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.430910    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpfw\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431045 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.430957    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431045 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431030    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431060    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431106    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431145    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431315    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431366    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431434    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.431778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431712    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.432029 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.431864    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.432103 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.432081    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.432477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.432460    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.433153 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.433132    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.434034 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.434011    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.434984 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.434961    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.495934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.495900    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpfw\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.496280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.495907    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.554619 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.554494    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:09.638075 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.638044    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"]
Nov 03 17:57:09.647928 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.647466    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.647928 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.647554    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.647928 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.647597    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.685980 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.685944    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"]
Nov 03 17:57:09.729349 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.729295    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.733662 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.732791    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.733662 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:09.732893    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:10.014186 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:10.014143    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"]
Nov 03 17:57:10.074869 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:57:10.074774    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78778b6_1409_4437_b7eb_8de9aea31362.slice/crio-f1f37b590cb6dcecd64af3886207a8854744c346ba53cbcd1f17a5444fc01930 WatchSource:0}: Error finding container f1f37b590cb6dcecd64af3886207a8854744c346ba53cbcd1f17a5444fc01930: Status 404 returned error can't find the container with id f1f37b590cb6dcecd64af3886207a8854744c346ba53cbcd1f17a5444fc01930
Nov 03 17:57:10.090940 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:10.090871    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:10.091330 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:10.091260    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:10.091600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:10.091573    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:11.071760 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:11.071129    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:57:11.102674 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.102633    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" event={"ID":"ddace43b-4112-4594-92df-095167b2bb3f","Type":"ContainerStarted","Data":"61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34"}
Nov 03 17:57:11.102961 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.102940    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" event={"ID":"ddace43b-4112-4594-92df-095167b2bb3f","Type":"ContainerStarted","Data":"8861b75c6b997a1ae4d2b93bcb9d8c8123969c9f794b726bee143c47d3cffda4"}
Nov 03 17:57:11.149786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.149720    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" event={"ID":"a78778b6-1409-4437-b7eb-8de9aea31362","Type":"ContainerStarted","Data":"f1f37b590cb6dcecd64af3886207a8854744c346ba53cbcd1f17a5444fc01930"}
Nov 03 17:57:11.175113 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.172223    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" event={"ID":"916ebdff-e9a4-47b1-ab2a-779238cd7430","Type":"ContainerStarted","Data":"c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107"}
Nov 03 17:57:11.175113 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.172298    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" event={"ID":"916ebdff-e9a4-47b1-ab2a-779238cd7430","Type":"ContainerStarted","Data":"a7d4c3313043463f47471cc21f8b2a4277dd041612e58a95a655ab8cfdace28a"}
Nov 03 17:57:11.232691 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.232583    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" event={"ID":"c86aa080-6e71-4065-a5c6-a6763a64541b","Type":"ContainerStarted","Data":"53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c"}
Nov 03 17:57:11.232996 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.232967    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" event={"ID":"c86aa080-6e71-4065-a5c6-a6763a64541b","Type":"ContainerStarted","Data":"6438394e1cecc5c051d1cbecee807516fcc8fa8cf8dbe9260477cafe892ca639"}
Nov 03 17:57:11.250949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.250822    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:11.269411 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.269370    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.269720 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.269690    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:11.307885 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.307840    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" event={"ID":"cee56aec-eddf-4726-8ab6-b3a71133cf69","Type":"ContainerStarted","Data":"4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24"}
Nov 03 17:57:11.308180 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.308156    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" event={"ID":"cee56aec-eddf-4726-8ab6-b3a71133cf69","Type":"ContainerStarted","Data":"294beee60fd30c85d9aeff98119c08f19519f4c4f7f938f25176fef0b3718c99"}
Nov 03 17:57:11.383770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.364871    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" event={"ID":"5c185a67-f484-4c14-9e8c-f560fb6c0db9","Type":"ContainerStarted","Data":"86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416"}
Nov 03 17:57:11.383770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.364943    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" event={"ID":"5c185a67-f484-4c14-9e8c-f560fb6c0db9","Type":"ContainerStarted","Data":"4def8c664c834720d188c8d95fc91d2af0457e86a73b712148e303872bc612eb"}
Nov 03 17:57:11.410230 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.410119    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podStartSLOduration=3.41009233 podStartE2EDuration="3.41009233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.270474476 +0000 UTC m=+1367.281212565" watchObservedRunningTime="2026-11-03 17:57:11.41009233 +0000 UTC m=+1367.420830410"
Nov 03 17:57:11.524313 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.524183    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podStartSLOduration=3.524155964 podStartE2EDuration="3.524155964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.397258527 +0000 UTC m=+1367.407996626" watchObservedRunningTime="2026-11-03 17:57:11.524155964 +0000 UTC m=+1367.534894037"
Nov 03 17:57:11.585757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.585535    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:11.611760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.591193    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podStartSLOduration=3.5911682579999997 podStartE2EDuration="3.591168258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.566587407 +0000 UTC m=+1367.577325539" watchObservedRunningTime="2026-11-03 17:57:11.591168258 +0000 UTC m=+1367.601906339"
Nov 03 17:57:11.817847 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.814390    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:11.817847 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.814514    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:11.835778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.830391    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:11.835778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.831221    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.835778 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.831272    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:11.927773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.925915    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:11.953758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.939078    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.953758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.939141    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:11.966199 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.966163    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:11.980905 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.980867    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.981990 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.981957    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:11.982250 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:11.980850    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podStartSLOduration=3.9808259440000002 podStartE2EDuration="3.980825944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.836641024 +0000 UTC m=+1367.847379088" watchObservedRunningTime="2026-11-03 17:57:11.980825944 +0000 UTC m=+1367.991564016"
Nov 03 17:57:12.002350 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.002283    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podStartSLOduration=4.002258929 podStartE2EDuration="4.002258929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.982640096 +0000 UTC m=+1367.993378161" watchObservedRunningTime="2026-11-03 17:57:12.002258929 +0000 UTC m=+1368.012997002"
Nov 03 17:57:12.013140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.013091    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.174773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.173481    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.227760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.227183    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.227760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.227281    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:12.251761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.229844    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.251761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.229907    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:12.305770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.295661    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:12.437590 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.437483    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.607823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.593169    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.607823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.593711    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:57:12.834831 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.664630    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.898787 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.868141    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.898787 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.893699    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.898787 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.895103    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:57:12.898787 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.895185    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:57:13.012503 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:12.989679    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.272758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.269685    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.272758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.269783    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:13.272758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.270497    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.272758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.270833    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:57:13.272758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.176014    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:13.272758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.272597    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:13.295314 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.120888    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:13.567821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.561843    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.567821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.563004    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.567821 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.563053    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:13.578760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.573949    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.588010 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.587973    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:57:13.758759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.709131    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.758759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.709190    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:13.758759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.709336    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" event={"ID":"a78778b6-1409-4437-b7eb-8de9aea31362","Type":"ContainerStarted","Data":"80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd"}
Nov 03 17:57:13.771685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.768859    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.827600 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.827382    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.828008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.827864    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:13.834786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.831548    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.834786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.831619    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:13.919296 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.900838    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.988352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:13.988299    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:14.193949 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.045232    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.215965 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.045274    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.216296 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.084270    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.216525 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.216481    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:14.302041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.284833    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.302041 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.285456    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.532820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.499782    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.754761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.702640    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:14.754761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.533959    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.754761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.702779    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:14.754761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.679023    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.754761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.702847    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:14.803100 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.802985    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.957762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.909898    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.957762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.909984    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:14.957762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.913404    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.957762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.913475    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:14.957762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.937621    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.957762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.937687    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:14.961945 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.961894    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.027760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:14.978951    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.027760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.002057    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:15.438759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.413516    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.424654    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.618834    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.570556    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.570590    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.525808    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.568279    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.618936    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.571686    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.623176 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.618968    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:15.839524 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.839419    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.935997 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.922884    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.978879 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.978839    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.979138 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:15.979107    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:16.014113 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.014079    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.014345 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.014321    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:16.065792 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.054586    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.065792 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.054651    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:16.138877 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:16.126852    5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.002s"
Nov 03 17:57:16.138877 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.126987    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:16.282760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.277566    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.282760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.277628    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:16.286758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.284852    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.469818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.468711    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.469818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.468801    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:16.572440 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.572402    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.576000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.575961    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:16.650886 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.594381    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podStartSLOduration=7.594359726 podStartE2EDuration="7.594359726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:16.283670731 +0000 UTC m=+1372.294408805" watchObservedRunningTime="2026-11-03 17:57:16.594359726 +0000 UTC m=+1372.605097801"
Nov 03 17:57:16.790972 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.790866    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.791226 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.791194    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:16.850388 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:16.847825    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.002848 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.002798    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.003441 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.003412    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.003869 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.003846    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.004265 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.004242    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.045842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.038876    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.045842 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.038942    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:17.081254 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.080551    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.081254 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.080611    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:17.230073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.230034    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.233032 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.232983    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:17.258850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.231185    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.259134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.259102    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:17.298073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.297955    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.351759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.326564    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.564629 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.564530    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.585766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.565187    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:17.787760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.787125    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.787760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.787191    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:17.849837 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.849702    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.908147 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.908109    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.908479 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.908414    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:17.971054 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.971019    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.971352 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:17.971324    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:18.008000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.004840    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.008000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.005473    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.008000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.005714    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.008000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.005989    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.220759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.218605    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.220759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.218665    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:18.237759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.235597    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.237759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.235657    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:18.298910 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.298847    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.555314 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.555214    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.555572 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.555544    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:18.794154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.794115    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.794872 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.794798    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:18.889211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.888828    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.923781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.923207    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.923781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.923284    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:18.977480 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.977353    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.977480 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:18.977417    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:19.045511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.043625    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.045511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.044034    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded"
Nov 03 17:57:19.045511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.044105    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.045511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.044363    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.045511 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.044640    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.274555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.213410    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:19.274555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.216166    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.274555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.216221    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:19.274555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.224240    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.274555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.224292    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:19.299858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.299813    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.556828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.556177    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.556828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.556240    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:19.801761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.800696    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.801761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.800792    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:19.900864 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.900809    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.947766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.914112    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.954890 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:19.954839    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:20.050913 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.050862    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:20.067770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.051824    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:20.067770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.052175    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:20.067770 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.052449    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:20.075988 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.075949    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.076258 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.076231    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:20.243822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.219581    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.244134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.244096    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:20.262935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.262870    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.263520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.263292    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:20.365392 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.365332    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:20.556939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.556843    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.557195 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.557162    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:20.809822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.805873    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.834840 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.822821    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:20.938069 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.936028    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.938069 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.936096    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:21.001125 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.974130    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:21.001125 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.996286    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:21.001125 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:20.996359    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:21.070823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.055928    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:21.070823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.056824    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:21.070823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.062867    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:21.070823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.063816    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:21.352183 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.316037    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:21.432695 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.432651    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:21.432941 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.379564    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:21.458867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.329357    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:21.458867 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.434847    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:21.792759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.774797    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:21.804794 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.793912    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:21.804794 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.801015    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:21.804794 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.801073    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:21.998762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:21.994951    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:22.072820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.070019    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.072820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.070109    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 17:57:22.072820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.072616    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.072820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.072670    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 17:57:22.072820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.072793    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:22.211760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.184794    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:22.300149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.212111    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:22.366438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.365061    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.366438 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.365120    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:22.433236 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.433194    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:22.712755 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.705103    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.712755 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.705186    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:22.724994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.724946    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.732864 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.732815    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:22.733105 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.727462    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:22.768765 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.751887    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:22.824479 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.802091    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:22.824479 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.805612    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.824479 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:22.805702    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 17:57:23.017059 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.016922    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.076007 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.075939    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:23.254277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.254240    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 17:57:23.254594 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.254555    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 17:57:23.257466 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.257432    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:23.289548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.289444    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 17:57:23.289895 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.289865    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.318762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.314383    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.446616 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.446567    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.572822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.567158    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:23.572822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.567223    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:23.623093 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.623050    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:23.737861 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.736870    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.753941 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.753830    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.794316 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.794282    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:23.927781 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.927750    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:23.962647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.962608    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:23.963082 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.963037    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:23.963661 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:23.962987    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:24.031230 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.031158    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:24.048888 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.046979    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:24.069131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.069087    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:24.069436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.069407    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:24.082769 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.082704    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:24.373819 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.354823    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:24.494864 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.494817    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:24.547433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.542437    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:24.547433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.542494    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:24.547433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.542543    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:57:24.547433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.542562    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:24.547433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.542606    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:57:24.547433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.542665    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:24.557375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.555480    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:24.557375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.556888    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:24.557375 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.556973    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:24.572554 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.572508    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 17:57:24.574462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.572771    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 17:57:24.574462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.572994    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:57:24.575057 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.574980    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 17:57:24.581370 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.581345    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:57:24.593404 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.593370    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:24.606286 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:24.606257    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:57:25.065754 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:25.065124    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:25.112846 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:25.112809    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:25.559906 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:25.559795    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:25.561421 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:25.560416    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 17:57:25.572539 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:25.572483    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 17:57:25.572972 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:25.572951    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 17:57:26.043775 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:26.041296    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:57:26.560335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:26.560299    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:26.561495 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:26.561474    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 17:57:28.247394 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:28.246114    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:28.288904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:28.288857    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.320547 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:28.320500    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.853221 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:28.852823    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.017760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.017683    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.025791 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.017603    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.078760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.077841    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.266060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.266012    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.267118 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.267093    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:57:29.293884 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.293830    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.294601 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.294572    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:29.336804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.335372    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.336804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.335838    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:57:29.889832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.863977    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:29.889832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:29.872138    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:57:30.042762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.031771    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:30.142629 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.079836    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.262433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.262295    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:30.262433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.262360    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.295858 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.285028    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.297216 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.296592    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.478722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.478623    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:57:30.501342 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.490644    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:57:30.519174 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.519124    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.678759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.677979    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:30.678759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.678147    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.883540 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:30.883415    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.201773 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:31.182960    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.214788 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:31.208016    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.345636 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:31.328504    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.399831 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:31.366856    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.581574 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:31.565260    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:32.047761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.035266    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.272436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.272372    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:32.272794 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.272760    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.308822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.302287    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.380039 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.379988    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:32.394762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.387846    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.446379 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.446333    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:32.449125 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.449086    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:32.689647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.452171    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:32.762759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.452505    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.770225 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.572462    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:32.799763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.773870    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.815303 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.631992    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.815588 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.801699    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:32.848761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:32.844337    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:57:33.076814 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.064851    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:33.489761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.439833    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:33.490843 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.472860    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:33.762614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.762401    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:33.772935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.772884    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:33.795767 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.794991    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Nov 03 17:57:33.990301 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:33.990267    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:34.070256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:34.069869    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:34.530903 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:34.530856    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:34.595698 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:34.595667    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:57:34.596137 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:34.596118    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:57:34.606889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:34.606616    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:57:34.656583 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:34.656549    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:57:37.926066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:37.926032    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:38.330512 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:38.330406    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:39.278204 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:39.278165    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"]
Nov 03 17:57:39.278710 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:39.278448    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" containerID="cri-o://d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8" gracePeriod=30
Nov 03 17:57:40.342485 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:40.342439    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:57:41.258244 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.258202    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7213-b79a15"]
Nov 03 17:57:41.263931 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.263900    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:41.315258 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.315224    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7213-b79a15"]
Nov 03 17:57:41.360807 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.360689    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6hkxc\""
Nov 03 17:57:41.423365 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.423328    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klb4k\" (UniqueName: \"kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k\") pod \"test-curl-7213-b79a15\" (UID: \"6843cf74-1fc4-4db8-96e0-fb57957e4d09\") " pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:41.525239 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.525143    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klb4k\" (UniqueName: \"kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k\") pod \"test-curl-7213-b79a15\" (UID: \"6843cf74-1fc4-4db8-96e0-fb57957e4d09\") " pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:41.589607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.589569    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klb4k\" (UniqueName: \"kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k\") pod \"test-curl-7213-b79a15\" (UID: \"6843cf74-1fc4-4db8-96e0-fb57957e4d09\") " pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:41.879823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:41.879719    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:42.266640 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:42.266593    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7213-b79a15"]
Nov 03 17:57:42.267018 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:57:42.266983    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6843cf74_1fc4_4db8_96e0_fb57957e4d09.slice/crio-3686e5009e84eb6a106550fa124da4a226a2d24d1c7b1d8964fbd0ec76d61ea6 WatchSource:0}: Error finding container 3686e5009e84eb6a106550fa124da4a226a2d24d1c7b1d8964fbd0ec76d61ea6: Status 404 returned error can't find the container with id 3686e5009e84eb6a106550fa124da4a226a2d24d1c7b1d8964fbd0ec76d61ea6
Nov 03 17:57:42.269140 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:42.269124    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:57:42.684168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:42.684136    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7213-b79a15" event={"ID":"6843cf74-1fc4-4db8-96e0-fb57957e4d09","Type":"ContainerStarted","Data":"3686e5009e84eb6a106550fa124da4a226a2d24d1c7b1d8964fbd0ec76d61ea6"}
Nov 03 17:57:43.136328 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:43.136219    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:43.288553 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:43.288515    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-5tvsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:57:43.288836 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:43.288587    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:57:44.629652 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.629611    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:57:44.674503 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674471    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.674702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674522    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.674702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674688    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.674877 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674803    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:44.674877 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674830    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.674877 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674853    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.675038 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674955    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.675038 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.674991    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.675148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.675081    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:44.675148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.675105    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.675244 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.675167    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket\") pod \"4ed72a79-ce64-41f0-b186-56b6a94821ef\" (UID: \"4ed72a79-ce64-41f0-b186-56b6a94821ef\") "
Nov 03 17:57:44.675244 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.675136    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:44.675418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.675395    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:44.675474 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.675433    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:44.676156 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.676128    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.676300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.676159    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.676300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.676176    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.676300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.676191    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.676300 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.676206    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed72a79-ce64-41f0-b186-56b6a94821ef-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.677106 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.677084    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "kube-api-access-6rc9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:44.677446 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.677430    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:44.677643 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.677628    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:57:44.677798 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.677780    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token" pod "4ed72a79-ce64-41f0-b186-56b6a94821ef" (UID: "4ed72a79-ce64-41f0-b186-56b6a94821ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:44.702066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.702026    5838 generic.go:356] "Generic (PLEG): container finished" podID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerID="d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8" exitCode=0
Nov 03 17:57:44.702256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.702083    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"
Nov 03 17:57:44.702256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.702181    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" event={"ID":"4ed72a79-ce64-41f0-b186-56b6a94821ef","Type":"ContainerDied","Data":"d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8"}
Nov 03 17:57:44.702256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.702227    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl" event={"ID":"4ed72a79-ce64-41f0-b186-56b6a94821ef","Type":"ContainerDied","Data":"79a0e4f3b642fee8e4353c317963979acc89b553c5342ab68bae1f2937e44694"}
Nov 03 17:57:44.702256 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.702256    5838 scope.go:122] "RemoveContainer" containerID="d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8"
Nov 03 17:57:44.727293 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.727255    5838 scope.go:122] "RemoveContainer" containerID="d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8"
Nov 03 17:57:44.727616 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:44.727550    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8\": container with ID starting with d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8 not found: ID does not exist" containerID="d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8"
Nov 03 17:57:44.727616 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.727585    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8"} err="failed to get container status \"d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8\": rpc error: code = NotFound desc = could not find container \"d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8\": container with ID starting with d499d2e15c16ba8fef0d55a2981cc9acc27762ff2bb3972a5f686de6b5cc84e8 not found: ID does not exist"
Nov 03 17:57:44.777332 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.777298    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rc9c\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-kube-api-access-6rc9c\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.777332 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.777333    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.777594 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.777345    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.777594 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.777354    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed72a79-ce64-41f0-b186-56b6a94821ef-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:44.929572 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.929532    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"]
Nov 03 17:57:44.934440 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:44.934413    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-5tvsl"]
Nov 03 17:57:45.714025 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:45.713979    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7213-b79a15" event={"ID":"6843cf74-1fc4-4db8-96e0-fb57957e4d09","Type":"ContainerStarted","Data":"44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b"}
Nov 03 17:57:45.833863 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:45.833708    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7213-b79a15" podStartSLOduration=1.587385351 podStartE2EDuration="4.833692658s" totalImagesPullingTime="3.246307307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:45.831063731 +0000 UTC m=+1401.841801822" watchObservedRunningTime="2026-11-03 17:57:45.833692658 +0000 UTC m=+1401.844430729"
Nov 03 17:57:46.122912 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:46.122825    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" path="/var/lib/kubelet/pods/4ed72a79-ce64-41f0-b186-56b6a94821ef/volumes"
Nov 03 17:57:47.349059 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.349025    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7213-b79a15"]
Nov 03 17:57:47.357131 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.357097    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7213-b79a15"]
Nov 03 17:57:47.539623 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.539580    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7213-bf1d40"]
Nov 03 17:57:47.540331 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.540313    5838 cpu_manager.go:414] "containerMap: removing container" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy"
Nov 03 17:57:47.540331 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.540332    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy"
Nov 03 17:57:47.540518 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.540353    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" containerName="test-curl-7213-b79a15"
Nov 03 17:57:47.540518 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.540358    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" containerName="test-curl-7213-b79a15"
Nov 03 17:57:47.540518 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.540514    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed72a79-ce64-41f0-b186-56b6a94821ef" containerName="istio-proxy"
Nov 03 17:57:47.540665 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.540528    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" containerName="test-curl-7213-b79a15"
Nov 03 17:57:47.544822 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.544800    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:47.601475 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.601392    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7213-bf1d40"]
Nov 03 17:57:47.622060 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.622022    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgrr\" (UniqueName: \"kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr\") pod \"test-curl-7213-bf1d40\" (UID: \"18fb25fb-f6f3-4d59-822a-052d710e86cb\") " pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:47.723508 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.723458    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgrr\" (UniqueName: \"kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr\") pod \"test-curl-7213-bf1d40\" (UID: \"18fb25fb-f6f3-4d59-822a-052d710e86cb\") " pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:47.734130 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.734088    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7213-b79a15" podUID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" containerName="test-curl-7213-b79a15" containerID="cri-o://44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b" gracePeriod=2
Nov 03 17:57:47.785810 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:47.785772    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7213-b79a15\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:47.787223 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.787199    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgrr\" (UniqueName: \"kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr\") pod \"test-curl-7213-bf1d40\" (UID: \"18fb25fb-f6f3-4d59-822a-052d710e86cb\") " pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:47.858542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:47.858448    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:48.257573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:48.257540    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7213-bf1d40"]
Nov 03 17:57:48.258415 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:57:48.258383    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18fb25fb_f6f3_4d59_822a_052d710e86cb.slice/crio-58ea7342b8f657c1287cc42350d23285b8b3624c3d029b2a5daa45f31b2e8abb WatchSource:0}: Error finding container 58ea7342b8f657c1287cc42350d23285b8b3624c3d029b2a5daa45f31b2e8abb: Status 404 returned error can't find the container with id 58ea7342b8f657c1287cc42350d23285b8b3624c3d029b2a5daa45f31b2e8abb
Nov 03 17:57:48.320211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:48.320170    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:48.750136 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:48.749421    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7213-bf1d40" event={"ID":"18fb25fb-f6f3-4d59-822a-052d710e86cb","Type":"ContainerStarted","Data":"85f3b2f7ef9b3fd15b664482367410e1da43f649098b5ed0a8b3a85d6e20bf87"}
Nov 03 17:57:48.750136 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:48.749499    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7213-bf1d40" event={"ID":"18fb25fb-f6f3-4d59-822a-052d710e86cb","Type":"ContainerStarted","Data":"58ea7342b8f657c1287cc42350d23285b8b3624c3d029b2a5daa45f31b2e8abb"}
Nov 03 17:57:48.870006 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:48.869955    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7213-bf1d40" podStartSLOduration=1.8699408499999999 podStartE2EDuration="1.86994085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:48.866621859 +0000 UTC m=+1404.877359946" watchObservedRunningTime="2026-11-03 17:57:48.86994085 +0000 UTC m=+1404.880678922"
Nov 03 17:57:50.026915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.026880    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:50.061871 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.061841    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k\" (UniqueName: \"kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k\") pod \"6843cf74-1fc4-4db8-96e0-fb57957e4d09\" (UID: \"6843cf74-1fc4-4db8-96e0-fb57957e4d09\") "
Nov 03 17:57:50.064162 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.064136    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k" pod "6843cf74-1fc4-4db8-96e0-fb57957e4d09" (UID: "6843cf74-1fc4-4db8-96e0-fb57957e4d09"). InnerVolumeSpecName "kube-api-access-klb4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:50.119151 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.119115    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" path="/var/lib/kubelet/pods/6843cf74-1fc4-4db8-96e0-fb57957e4d09/volumes"
Nov 03 17:57:50.163592 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.163559    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klb4k\" (UniqueName: \"kubernetes.io/projected/6843cf74-1fc4-4db8-96e0-fb57957e4d09-kube-api-access-klb4k\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:50.483689 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.483638    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7213-bf1d40"]
Nov 03 17:57:50.491490 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.491452    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7213-bf1d40"]
Nov 03 17:57:50.768396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.768300    5838 generic.go:356] "Generic (PLEG): container finished" podID="6843cf74-1fc4-4db8-96e0-fb57957e4d09" containerID="44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b" exitCode=137
Nov 03 17:57:50.768396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.768346    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-b79a15"
Nov 03 17:57:50.768625 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.768408    5838 scope.go:122] "RemoveContainer" containerID="44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b"
Nov 03 17:57:50.768689 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.768656    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7213-bf1d40" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" containerName="test-curl-7213-bf1d40" containerID="cri-o://85f3b2f7ef9b3fd15b664482367410e1da43f649098b5ed0a8b3a85d6e20bf87" gracePeriod=2
Nov 03 17:57:50.779456 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.779433    5838 scope.go:122] "RemoveContainer" containerID="44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b"
Nov 03 17:57:50.779846 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:50.779824    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b\": container with ID starting with 44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b not found: ID does not exist" containerID="44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b"
Nov 03 17:57:50.779922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:50.779853    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b"} err="failed to get container status \"44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b\": rpc error: code = NotFound desc = could not find container \"44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b\": container with ID starting with 44d4f8adf0d040a975e97eabc8322d9bd76c8e583f07c6728321289d55e58b5b not found: ID does not exist"
Nov 03 17:57:50.820304 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:50.820243    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7213-bf1d40\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:50.872476 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:50.872418    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7213-bf1d40\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:57:51.221695 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:51.221664    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:51.379657 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:51.379619    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:51.828604 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:51.828570    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:52.000157 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:51.997090    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:52.169758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:52.163448    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:52.999793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:52.997372    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"]
Nov 03 17:57:52.999793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:52.998475    5838 cpu_manager.go:414] "containerMap: removing container" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" containerName="test-curl-7213-bf1d40"
Nov 03 17:57:52.999793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:52.998498    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" containerName="test-curl-7213-bf1d40"
Nov 03 17:57:52.999793 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:52.998702    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" containerName="test-curl-7213-bf1d40"
Nov 03 17:57:53.042134 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.041166    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.100844 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.079160    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"]
Nov 03 17:57:53.222266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.117624    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c74b2c26-gw-openshift-default-dockercfg-6v4zv\""
Nov 03 17:57:53.222266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.117897    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387235    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387307    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387343    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7knj\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387393    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387535    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387594    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387655    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387688    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.433805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.387748    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.657168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.554085    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.657168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.555650    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:53.657841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.657805    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.658035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.658016    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7knj\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.658181 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.658165    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.658477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.658453    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.658666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.658646    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.658861 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.658841    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.659011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.658992    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.659150 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.659135    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.659373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.659356    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.669253    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.670488    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.670790    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.674458    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.674765    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.689539    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.764199    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.816485 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.802401    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.868570 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.867310    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:53.909823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.888872    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.909823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:53.903517    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7knj\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj\") pod \"e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") " pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:54.020328 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.020246    5838 generic.go:356] "Generic (PLEG): container finished" podID="18fb25fb-f6f3-4d59-822a-052d710e86cb" containerID="85f3b2f7ef9b3fd15b664482367410e1da43f649098b5ed0a8b3a85d6e20bf87" exitCode=137
Nov 03 17:57:54.164330 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.164027    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:57:54.296627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.296573    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:54.613008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.607825    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.628979 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.623430    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:54.652009 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.651181    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.676792 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.672886    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:57:54.676792 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.672952    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:57:54.811765 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.809187    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.811765 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.809656    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:54.916940 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.912799    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.980901 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:54.935850    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:57:55.329103 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:55.329058    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.379913 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:55.355094    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:57:55.466484 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:55.466446    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"]
Nov 03 17:57:55.708486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:55.697125    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:55.765761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:55.760065    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:55.765761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:55.760517    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:56.086764 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.029257    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:56.143760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.122825    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.197765 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.067871    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:56.232767 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.198816    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.490160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.477051    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:56.585498 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.584859    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:56.585498 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.584925    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.710265 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.710234    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:57:56.710566 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.710548    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:56.710758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.710719    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:57:56.711083 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.711063    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"]
Nov 03 17:57:56.829004 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.828883    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.397873 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.878395    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:57.397873 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:56.974467    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.397873 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.065191    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.397873 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:57.285632    5838 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18fb25fb_f6f3_4d59_822a_052d710e86cb.slice/crio-85f3b2f7ef9b3fd15b664482367410e1da43f649098b5ed0a8b3a85d6e20bf87.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18fb25fb_f6f3_4d59_822a_052d710e86cb.slice/crio-conmon-85f3b2f7ef9b3fd15b664482367410e1da43f649098b5ed0a8b3a85d6e20bf87.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 17:57:57.419259 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.326271    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:57.419259 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.403149    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.505790 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.495846    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.606952 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.606898    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.643390 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.643351    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:57.643693 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.643664    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.960757 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:57.956795    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:58.198056 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.198005    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qrswt" podUID="f956514d-9307-413e-a516-7d5449040eb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.468762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.408060    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:58.496948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.410230    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.497387 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.430040    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:58.785492 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.785388    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.811832 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.810342    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:58.812308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.810396    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:57:58.902615 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.902572    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.908890 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:58.908822    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:57:59.144107 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.144066    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.144484 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.144446    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311181    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhgt\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311434 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311289    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311434 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311328    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311434 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311374    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311611 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311472    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311683 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311663    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.311808 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.311788    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.315800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.224144    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 17:57:59.374760 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:57:59.345213    5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.634s"
Nov 03 17:57:59.459968 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.459879    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.405709    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.532404    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619249    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619303    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619396    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619558    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619646    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619772    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619801    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.619862    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhgt\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.607233    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.635779    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.608962    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:57:59.668817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.636977    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.689982 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.678902    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.689982 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.680072    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.719002 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.699398    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.747433 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.747395    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:57:59.857841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.851245    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:59.857841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.851451    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:59.857841 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.851523    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:59.917759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.916921    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:59.917759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.917348    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:57:59.917759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.917647    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:57:59.918088 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:57:59.917921    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 17:58:00.257190 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.257097    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:58:00.317716 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.317685    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:00.318373 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.315307    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr\" (UniqueName: \"kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr\") pod \"18fb25fb-f6f3-4d59-822a-052d710e86cb\" (UID: \"18fb25fb-f6f3-4d59-822a-052d710e86cb\") "
Nov 03 17:58:00.403994 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.403955    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:00.484928 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:58:00.484886    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7213-bf1d40\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:58:00.563209 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.563128    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:58:00.567888 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.567859    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"]
Nov 03 17:58:00.588243 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.563410    5838 scope.go:122] "RemoveContainer" containerID="85f3b2f7ef9b3fd15b664482367410e1da43f649098b5ed0a8b3a85d6e20bf87"
Nov 03 17:58:00.621860 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.621818    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:00.637701 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.637658    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:00.667774 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.641496    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhgt\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:00.782993 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.782952    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr" pod "18fb25fb-f6f3-4d59-822a-052d710e86cb" (UID: "18fb25fb-f6f3-4d59-822a-052d710e86cb"). InnerVolumeSpecName "kube-api-access-flgrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:00.789465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.789208    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:58:00.789465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.789267    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:58:00.789465 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.789299    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:58:00.791935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.791905    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:58:00.818836 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.803801    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:58:00.822508 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.822486    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:58:00.854291 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.854255    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgrr\" (UniqueName: \"kubernetes.io/projected/18fb25fb-f6f3-4d59-822a-052d710e86cb-kube-api-access-flgrr\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:00.902142 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:00.902096    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:01.185420 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.185389    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:58:01.187567 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.187532    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" event={"ID":"d8babccd-6177-41a8-91c3-6cb2b4d52032","Type":"ContainerStarted","Data":"df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"}
Nov 03 17:58:01.187785 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.187602    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" event={"ID":"d8babccd-6177-41a8-91c3-6cb2b4d52032","Type":"ContainerStarted","Data":"587c55d51beccf3fb27da61cfb6f50edf01c3b0983ded112b4fe438b20c4f197"}
Nov 03 17:58:01.380152 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.380113    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"]
Nov 03 17:58:01.381238 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:58:01.381206    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a026b47_8024_43da_a7aa_7669e1712b0b.slice/crio-c2a41796af640339e631c2c8994e9e8b37f46505ee69b0a0db47c7bc68e59cdc WatchSource:0}: Error finding container c2a41796af640339e631c2c8994e9e8b37f46505ee69b0a0db47c7bc68e59cdc: Status 404 returned error can't find the container with id c2a41796af640339e631c2c8994e9e8b37f46505ee69b0a0db47c7bc68e59cdc
Nov 03 17:58:01.384526 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.384488    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:01.384627 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.384581    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:01.384701 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.384627    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:01.471916 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:58:01.471849    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7213-bf1d40\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" pod="opendatahub/test-curl-7213-bf1d40"
Nov 03 17:58:01.597270 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.597210    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podStartSLOduration=9.597190549 podStartE2EDuration="9.597190549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:01.597036504 +0000 UTC m=+1417.607774567" watchObservedRunningTime="2026-11-03 17:58:01.597190549 +0000 UTC m=+1417.607928624"
Nov 03 17:58:01.916466 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:01.916428    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:02.083589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.083550    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:02.121612 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.121573    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18fb25fb-f6f3-4d59-822a-052d710e86cb" path="/var/lib/kubelet/pods/18fb25fb-f6f3-4d59-822a-052d710e86cb/volumes"
Nov 03 17:58:02.165250 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.165217    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:58:02.171234 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.171146    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:58:02.197934 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.197896    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" event={"ID":"6a026b47-8024-43da-a7aa-7669e1712b0b","Type":"ContainerStarted","Data":"bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac"}
Nov 03 17:58:02.198145 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.197958    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" event={"ID":"6a026b47-8024-43da-a7aa-7669e1712b0b","Type":"ContainerStarted","Data":"c2a41796af640339e631c2c8994e9e8b37f46505ee69b0a0db47c7bc68e59cdc"}
Nov 03 17:58:02.198473 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.198447    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:58:02.199557 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.199527    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:58:02.260035 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.259993    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:02.559823 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.559706    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podStartSLOduration=7.5596890850000005 podStartE2EDuration="7.559689085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:02.555747929 +0000 UTC m=+1418.566485999" watchObservedRunningTime="2026-11-03 17:58:02.559689085 +0000 UTC m=+1418.570427157"
Nov 03 17:58:02.908769 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.908025    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:02.908769 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.908218    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 17:58:02.908769 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:02.908263    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 17:58:03.233480 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:03.233391    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:03.529464 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:03.529372    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:03.905196 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:03.905157    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 17:58:03.905543 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:03.905509    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 17:58:04.191187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:04.191083    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:04.373828 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:04.372892    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:04.923309 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:04.923266    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 17:58:04.923678 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:04.923643    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 17:58:05.906320 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:05.906280    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 17:58:05.907011 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:05.906958    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 17:58:06.915212 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:06.915173    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 17:58:06.915908 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:06.915866    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 17:58:07.607952 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:07.607911    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:07.775468 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:07.775433    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:07.916758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:07.910242    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 17:58:07.916758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:07.910322    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 17:58:07.945855 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:07.945818    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:09.332288 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.332242    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:09.332705 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.332581    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:09.333411 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.333384    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 17:58:09.894935 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.894890    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:09.930147 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.930097    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"]
Nov 03 17:58:09.941380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.941344    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.948970 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.948519    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"]
Nov 03 17:58:09.957795 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.957757    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:09.978210 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.978172    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.978560 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.978537    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.978922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.978902    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.979386 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.979366    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.979650 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.979630    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.979896 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.979876    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.980211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.980106    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntx4\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.980211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.980157    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:09.980453 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.980351    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.006875 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:09.996216    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"]
Nov 03 17:58:10.021199 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.019199    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"]
Nov 03 17:58:10.081634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081600    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.081634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081641    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zs9t\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.081921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081674    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.081921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081697    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.081921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081721    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntx4\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.081921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081763    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.081921 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081785    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.081940    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082039    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082075    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082118    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082134    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082183    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082216    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082247    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082255    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082295    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082382    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082421    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082489    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.082681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082526    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.082681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.082554    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.083481 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.083450    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.085459 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.084863    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.085589 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.085567    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.145697 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.145612    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.145896 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.145788    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntx4\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.184118 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184081    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184137    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184228    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184253    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184335 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184323    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184564 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184408    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184564 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184544    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zs9t\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184715 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184683    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184800    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.184870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.184817    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.185111 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.185056    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.185242 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.185220    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.185477 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.185450    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.186598 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.186222    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.188308 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.188282    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.188744 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.188703    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.248971 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.248933    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zs9t\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.250261 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.250232    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.262681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.262651    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:10.305083 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.305031    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:10.367077 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:58:10.360320    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:58:10.715878 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.715849    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"]
Nov 03 17:58:10.716340 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:58:10.716315    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ffcb69_1f1c_4c43_b6f4_e9ec036feaf8.slice/crio-76f046d4d5454494de580ace8307304398588f4dce07cecc6580761176ab6ab8 WatchSource:0}: Error finding container 76f046d4d5454494de580ace8307304398588f4dce07cecc6580761176ab6ab8: Status 404 returned error can't find the container with id 76f046d4d5454494de580ace8307304398588f4dce07cecc6580761176ab6ab8
Nov 03 17:58:10.719695 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.719663    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.719800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.719783    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.719848 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.719824    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.768461 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.768434    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"]
Nov 03 17:58:10.776059 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:58:10.776032    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09369850_5e93_42a8_973d_383ba3e53363.slice/crio-59743312bca7eb237907161d3346887cdc2d8b83a057f6dd59771fbeb55a7d78 WatchSource:0}: Error finding container 59743312bca7eb237907161d3346887cdc2d8b83a057f6dd59771fbeb55a7d78: Status 404 returned error can't find the container with id 59743312bca7eb237907161d3346887cdc2d8b83a057f6dd59771fbeb55a7d78
Nov 03 17:58:10.779635 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.779589    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.779723 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.779690    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.779817 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:10.779760    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:11.318455 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:11.318413    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" event={"ID":"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8","Type":"ContainerStarted","Data":"e82df84a1dd7bc5eeed5d5a5f4ea5fcccb3fe1480ebeb1561babd5eeeb5c6ad3"}
Nov 03 17:58:11.318681 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:11.318479    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" event={"ID":"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8","Type":"ContainerStarted","Data":"76f046d4d5454494de580ace8307304398588f4dce07cecc6580761176ab6ab8"}
Nov 03 17:58:11.321141 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:11.320491    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" event={"ID":"09369850-5e93-42a8-973d-383ba3e53363","Type":"ContainerStarted","Data":"f54213ef7e5308e7447fefde9d0a97506fdf8d8081fb1687740dd5facc0999e5"}
Nov 03 17:58:11.321141 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:11.320556    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" event={"ID":"09369850-5e93-42a8-973d-383ba3e53363","Type":"ContainerStarted","Data":"59743312bca7eb237907161d3346887cdc2d8b83a057f6dd59771fbeb55a7d78"}
Nov 03 17:58:11.450800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:11.449025    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podStartSLOduration=2.44900437 podStartE2EDuration="2.44900437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:11.444104619 +0000 UTC m=+1427.454842691" watchObservedRunningTime="2026-11-03 17:58:11.44900437 +0000 UTC m=+1427.459742442"
Nov 03 17:58:11.584941 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:11.584777    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podStartSLOduration=2.58475153 podStartE2EDuration="2.58475153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:11.582372071 +0000 UTC m=+1427.593110145" watchObservedRunningTime="2026-11-03 17:58:11.58475153 +0000 UTC m=+1427.595489653"
Nov 03 17:58:12.263217 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.263174    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:12.264783 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.264759    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 17:58:12.264837 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.264814    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 17:58:12.317939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.317900    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:12.341972 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.341930    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:12.342266 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.342233    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:12.549820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.548906    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"]
Nov 03 17:58:12.558148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.558116    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.602764 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.598304    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"]
Nov 03 17:58:12.626759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.622003    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 17:58:12.648760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.641974    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 17:58:12.648760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.642057    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 17:58:12.648760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.642195    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-96fc4\""
Nov 03 17:58:12.670836 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.670802    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.671051 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.671037    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.671328 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.671280    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.671431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.671350    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfck\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.671431 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.671388    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.671652 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.671610    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.671702 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.671661    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774163 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774126    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774181    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774245    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774292    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774462 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774427    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774501 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774470    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfck\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.774532 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.774498    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.775148 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.775124    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.776977 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.776943    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.777231 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.777209    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.777519 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.777498    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.777614 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.777556    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.862777 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.862647    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfck\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.864203 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.864174    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6m8b4\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:12.872833 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:12.872808    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:13.263337 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:13.263293    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 17:58:13.263542 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:13.263354    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 17:58:13.315722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:13.314220    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:13.315722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:13.314283    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:13.383244 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:13.383104    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"]
Nov 03 17:58:13.385617 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:58:13.385587    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95322a2_de73_457e_baf4_293854dd05c1.slice/crio-4e919a6770cda06d767e5d05e1137c934c16c89be41eb3ead14736cb3e7088ee WatchSource:0}: Error finding container 4e919a6770cda06d767e5d05e1137c934c16c89be41eb3ead14736cb3e7088ee: Status 404 returned error can't find the container with id 4e919a6770cda06d767e5d05e1137c934c16c89be41eb3ead14736cb3e7088ee
Nov 03 17:58:14.263392 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:14.263352    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 17:58:14.265991 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:14.263411    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 17:58:14.313758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:14.312865    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:14.313758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:14.312934    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:14.371816 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:14.371709    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" event={"ID":"a95322a2-de73-457e-baf4-293854dd05c1","Type":"ContainerStarted","Data":"4e919a6770cda06d767e5d05e1137c934c16c89be41eb3ead14736cb3e7088ee"}
Nov 03 17:58:15.102759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.102116    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:15.263578 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.263537    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 17:58:15.264074 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.263603    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 17:58:15.269104 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.269077    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:15.312759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.309219    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:15.312759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.309286    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:15.443756 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.443468    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:15.624762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.623943    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:15.819897 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:15.816175    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:16.449850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:16.441691    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:16.449850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:16.441812    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:16.449850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:16.444987    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 17:58:16.449850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:16.445049    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 17:58:16.871819 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:16.866297    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:17.053909 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:17.053864    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:17.263318 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:17.263256    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 17:58:17.263540 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:17.263359    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 17:58:17.310421 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:17.310271    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:17.310421 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:17.310335    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:18.306316 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:18.306272    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 17:58:18.307038 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:18.306343    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 17:58:18.900868 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:18.898932    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:18.900868 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:18.900664    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:18.903670 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:18.903643    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 17:58:19.365014 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:19.363582    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:19.365497 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:19.365081    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:19.368705 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:19.368676    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 17:58:20.135930 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:20.135891    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:20.136075 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:20.135962    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:20.880168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:20.880135    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" event={"ID":"a95322a2-de73-457e-baf4-293854dd05c1","Type":"ContainerStarted","Data":"5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b"}
Nov 03 17:58:20.880617 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:20.880200    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:21.109076 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:21.109017    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" podStartSLOduration=2.36237979 podStartE2EDuration="9.108996116s" totalImagesPullingTime="6.746616326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:21.054639179 +0000 UTC m=+1437.065377314" watchObservedRunningTime="2026-11-03 17:58:21.108996116 +0000 UTC m=+1437.119734188"
Nov 03 17:58:21.342704 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:21.342651    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:21.891146 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:21.891109    5838 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6m8b4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:58:21.891645 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:21.891182    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" podUID="a95322a2-de73-457e-baf4-293854dd05c1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:58:23.324135 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:23.324100    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:23.565657 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:23.562001    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:24.890061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:24.890030    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 17:58:26.191810 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:26.191766    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:58:26.651299 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:26.651263    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:28.500901 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:28.500848    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.593549 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:28.593501    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.706966 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:28.706892    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.762471 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:28.762249    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.005759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.004849    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.379765 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.366857    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.392670 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.392619    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:58:29.516877 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.516823    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.745774 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.607867    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.745774 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.710941    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.749625 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.749589    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:58:29.749964 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.749944    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:58:29.769915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.767837    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.769915 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.768521    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:58:29.770647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:29.770620    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:58:30.399755 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:30.397240    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.404365 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:30.404309    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.778895 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:30.747432    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:58:30.870951 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:30.870911    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:58:30.904834 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:30.779512    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.100077 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.099961    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.219923 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.188852    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.222399 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.222361    5838 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:31.223248 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.222967    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.325950 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.222846    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.569761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.543914    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:31.569761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:31.546388    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.116800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.015779    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.148790 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.146151    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.296758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.295256    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj" podUID="00ed915e-64f7-4efd-981b-71d7c8756b48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.392759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.389367    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.527000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.435588    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.527000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.439406    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:32.527000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.439489    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.545224 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.545170    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.572555 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.572505    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.651969 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.651862    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.680761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.674129    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:32.680761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.674453    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.988013 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:32.981788    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.059760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.036132    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.190763 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.173357    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.418761 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.399619    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.466775 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.459676    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.560365 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.560322    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.587601 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.587559    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:33.632449 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.587935    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:58:33.658149 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:33.588374    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:34.105759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.080261    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.282920 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.282859    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.445514 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.423323    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:34.445514 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.423391    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.501766 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.498531    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:58:34.507270 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.506402    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.640237 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.640187    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:34.711061 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:34.710951    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:35.116898 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.114831    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:35.600097 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.599828    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:58:35.619885 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.619833    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:35.627440 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.620152    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 17:58:35.662207 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.661400    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:35.719938 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.719865    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 17:58:35.731548 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.731509    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:35.812430 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.812398    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 17:58:35.858321 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.858222    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:35.954557 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:35.954435    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 17:58:36.034337 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:36.016328    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 17:58:36.118850 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:36.104027    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:58:36.132685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:36.132608    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:58:36.132685 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:36.132676    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 17:58:36.540062 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:36.539985    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:37.198359 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:37.198318    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:37.440804 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:37.431636    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:38.682927 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:38.682887    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:40.344313 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:58:40.344267    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:58:41.225471 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:41.225428    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:41.390092 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:41.390054    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:41.549799 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:41.549687    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:46.941645 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:46.941589    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:48.045674 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:48.045629    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:48.262569 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:48.258907    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:52.563553 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:52.563516    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:52.728742 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:52.728702    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:52.882024 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:52.881945    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:55.384758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.383916    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:55.384758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.383998    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.443280 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.443230    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.459886 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.459832    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.721802 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.720875    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.780000 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.779963    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"]
Nov 03 17:58:55.798760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.787847    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.851932 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.851889    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.852249 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.852224    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.852442 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.852422    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.852639 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.852620    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fm2l\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.852805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.852786    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.852946 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.852930    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.853154 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.853137    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.853351 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.853334    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.853466 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.853449    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:55.927337 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.927171    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"]
Nov 03 17:58:55.992814 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:55.954826    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.089549 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.089109    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.089903 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.089873    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.090202 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.090180    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.091197 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.090383    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.091436 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.091418    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.091640 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.091621    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fm2l\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.091811 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.091781    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.092026 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.091999    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.119800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.014721    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.119800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.112259    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.119800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.112539    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.119800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.113895    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.119800 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.114424    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.139222 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.139176    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.219491 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.189685    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.221350 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.221308    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.223759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.223651    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fm2l\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-92968\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.449510 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.449455    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:56.888556 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.888474    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"]
Nov 03 17:58:56.926215 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:58:56.926182    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8321e086_def0_4963_9ef0_f7e5c180f7f5.slice/crio-5df3302f6e1dd8bd4453ea0621608d347050210434b7a93eaab6bb2c89da20dc WatchSource:0}: Error finding container 5df3302f6e1dd8bd4453ea0621608d347050210434b7a93eaab6bb2c89da20dc: Status 404 returned error can't find the container with id 5df3302f6e1dd8bd4453ea0621608d347050210434b7a93eaab6bb2c89da20dc
Nov 03 17:58:56.930405 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.930363    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:56.930535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.930459    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:56.930535 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:56.930502    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:57.244751 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:57.244698    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" event={"ID":"8321e086-def0-4963-9ef0-f7e5c180f7f5","Type":"ContainerStarted","Data":"24b57541fe10b24a36e02a941c71f1dc8beb6a34d6bcd136f2f45c5fde6b8e91"}
Nov 03 17:58:57.244948 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:57.244783    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" event={"ID":"8321e086-def0-4963-9ef0-f7e5c180f7f5","Type":"ContainerStarted","Data":"5df3302f6e1dd8bd4453ea0621608d347050210434b7a93eaab6bb2c89da20dc"}
Nov 03 17:58:57.378285 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:57.378227    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podStartSLOduration=2.378209278 podStartE2EDuration="2.378209278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:57.373746115 +0000 UTC m=+1473.384484225" watchObservedRunningTime="2026-11-03 17:58:57.378209278 +0000 UTC m=+1473.388947350"
Nov 03 17:58:57.450211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:57.450180    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:58:58.451820 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:58.451776    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:58:58.452331 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:58.451843    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 17:58:58.887187 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:58.887098    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:59.450197 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:59.450163    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:58:59.450380 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:58:59.450232    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 17:59:00.450066 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:00.450032    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:59:00.450536 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:00.450092    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 17:59:01.427995 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:01.427954    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:01.450179 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:01.450133    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:59:01.450672 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:01.450195    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 17:59:02.245758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:02.232680    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:02.450558 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:02.450123    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:59:02.450558 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:02.450185    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 17:59:02.671507 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:02.671477    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:03.450989 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:03.450949    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:59:03.451509 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:03.451020    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 17:59:03.763643 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:03.763549    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:03.936857 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:03.936818    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:04.157026 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.156989    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"]
Nov 03 17:59:04.162966 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.162937    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.228443 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.228397    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:04.269008 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.268971    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c1ea78c-serving-cert\""
Nov 03 17:59:04.283866 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.283834    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"]
Nov 03 17:59:04.290265 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.290213    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.290459 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.290275    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.290459 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.290409    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.290605 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.290555    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.392068 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.392023    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.392277 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.392148    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.392360 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.392292    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.392360 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.392319    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.392538 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:59:04.392507    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-serving-cert: secret "maas-api-e2e-ait-1c1ea78c-serving-cert" not found
Nov 03 17:59:04.392674 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:59:04.392597    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls podName:ae26ee52-f1bb-4b72-b15d-a7f84274ab55 nodeName:}" failed. No retries permitted until 2026-11-03 17:59:04.892573231 +0000 UTC m=+1480.903311298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls") pod "maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" (UID: "ae26ee52-f1bb-4b72-b15d-a7f84274ab55") : secret "maas-api-e2e-ait-1c1ea78c-serving-cert" not found
Nov 03 17:59:04.392843 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.392822    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.394398 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.394364    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:04.395624 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.395591    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.464531 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.464266    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.560192 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.560152    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:04.740933 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.740716    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:04.770979 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.770938    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:59:04.773325 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.773246    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:59:04.774813 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.774707    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 17:59:04.902164 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.902129    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:04.905317 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:04.905285    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\") pod \"maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:05.079014 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:05.078929    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:05.494483 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:59:05.494448    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae26ee52_f1bb_4b72_b15d_a7f84274ab55.slice/crio-5e7f7ec9781df6ce9361ba403495658d29ac0af712833e40b2eeceb50b85db83 WatchSource:0}: Error finding container 5e7f7ec9781df6ce9361ba403495658d29ac0af712833e40b2eeceb50b85db83: Status 404 returned error can't find the container with id 5e7f7ec9781df6ce9361ba403495658d29ac0af712833e40b2eeceb50b85db83
Nov 03 17:59:05.495109 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:05.495088    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"]
Nov 03 17:59:06.515418 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:06.515379    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" event={"ID":"ae26ee52-f1bb-4b72-b15d-a7f84274ab55","Type":"ContainerStarted","Data":"449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da"}
Nov 03 17:59:06.515973 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:06.515443    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" event={"ID":"ae26ee52-f1bb-4b72-b15d-a7f84274ab55","Type":"ContainerStarted","Data":"5e7f7ec9781df6ce9361ba403495658d29ac0af712833e40b2eeceb50b85db83"}
Nov 03 17:59:06.515973 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:06.515657    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:06.642116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:06.642052    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" podStartSLOduration=1.843428783 podStartE2EDuration="2.642032638s" totalImagesPullingTime="798.603855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:06.637722778 +0000 UTC m=+1482.648460851" watchObservedRunningTime="2026-11-03 17:59:06.642032638 +0000 UTC m=+1482.652770709"
Nov 03 17:59:06.889452 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:06.889408    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:07.081444 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:07.081353    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:07.241234 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:07.241192    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:10.066484 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.066438    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"]
Nov 03 17:59:10.071772 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.071704    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.125578 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.125530    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\""
Nov 03 17:59:10.189327 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.189290    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.189520 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.189333    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.189573 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.189504    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.189676 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.189656    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.189786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.189721    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.189922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.189901    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhjf\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.190211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.190189    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.190310 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.190243    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.190378 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.190333    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.233886 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.233848    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:10.291486 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291440    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"]
Nov 03 17:59:10.291653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291544    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.291653 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291598    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.291825 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291691    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhjf\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.291825 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291777    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.291825 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291818    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.291956 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.291907    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.292020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.292004    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.292062 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.292035    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.292171 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.292152    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.293021 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.292995    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.294252 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.294136    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.294252 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.294145    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.294505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.294483    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.294786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.294765    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.324281 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.323904    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.326870 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.326296    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.361446 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.361412    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.361607 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.361409    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhjf\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-82sdk\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.375374 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:59:10.375307    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:59:10.395057 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.395015    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:10.867395 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.867361    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"]
Nov 03 17:59:10.868945 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:59:10.868915    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31406d2d_f93b_4f1b_bbf9_2e4aea874e94.slice/crio-e3270902cb1ddadea0183bfa1f0596c75fbe5cf6c695db3fcebb49c74de822f6 WatchSource:0}: Error finding container e3270902cb1ddadea0183bfa1f0596c75fbe5cf6c695db3fcebb49c74de822f6: Status 404 returned error can't find the container with id e3270902cb1ddadea0183bfa1f0596c75fbe5cf6c695db3fcebb49c74de822f6
Nov 03 17:59:10.872116 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.872078    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:10.872211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.872156    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:10.872211 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:10.872187    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:11.574835 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:11.574792    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" event={"ID":"31406d2d-f93b-4f1b-bbf9-2e4aea874e94","Type":"ContainerStarted","Data":"564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680"}
Nov 03 17:59:11.575476 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:11.574868    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" event={"ID":"31406d2d-f93b-4f1b-bbf9-2e4aea874e94","Type":"ContainerStarted","Data":"e3270902cb1ddadea0183bfa1f0596c75fbe5cf6c695db3fcebb49c74de822f6"}
Nov 03 17:59:11.703923 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:11.703852    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podStartSLOduration=1.7038273579999998 podStartE2EDuration="1.703827358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:11.701328863 +0000 UTC m=+1487.712066939" watchObservedRunningTime="2026-11-03 17:59:11.703827358 +0000 UTC m=+1487.714565432"
Nov 03 17:59:12.114758 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:12.112248    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:12.275059 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:12.275021    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:12.412759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:12.407803    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:12.412759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:12.411868    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:12.412759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:12.411932    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:12.533281 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:12.533242    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 17:59:13.420762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:13.416864    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:13.420762 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:13.416929    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:14.416760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:14.411622    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:14.416760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:14.411687    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:15.422186 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:15.422146    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:15.427014 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:15.423140    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:15.481630 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:15.481592    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:15.729020 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:15.726426    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:15.892992 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:15.892698    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:16.414760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:16.410199    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:16.414760 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:16.410263    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:17.405926 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:17.405886    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:17.406332 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:17.405948    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:18.405892 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:18.405856    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:59:18.406073 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:18.405915    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 17:59:19.652618 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:19.652519    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:19.653185 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:19.653122    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:19.654505 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:19.654481    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 17:59:21.305160 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:21.305126    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:21.657147 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:21.657104    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:24.382982 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:24.382948    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:59:24.392904 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:24.392877    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 17:59:26.404713 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:26.404679    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:31.502001 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:31.501966    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:36.440889 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.440848    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"]
Nov 03 17:59:36.448090 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.448057    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.507213 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.507175    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"]
Nov 03 17:59:36.507528 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.507501    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.507678 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.507660    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.507816 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.507748    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhlb\" (UniqueName: \"kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.507907 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.507854    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.551396 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.551362    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-aa500779-serving-cert\""
Nov 03 17:59:36.609206 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.609167    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.609419 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.609306    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.609504 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.609485    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.609652 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.609633    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhlb\" (UniqueName: \"kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.609724 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:59:36.609653    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-serving-cert: secret "maas-api-e2e-derive-aa500779-serving-cert" not found
Nov 03 17:59:36.609811 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:59:36.609785    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls podName:321f7f0e-5d10-478a-9173-531b70d1c4db nodeName:}" failed. No retries permitted until 2026-11-03 17:59:37.109719473 +0000 UTC m=+1513.120457523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls") pod "maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" (UID: "321f7f0e-5d10-478a-9173-531b70d1c4db") : secret "maas-api-e2e-derive-aa500779-serving-cert" not found
Nov 03 17:59:36.610276 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.610253    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.612324 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.612297    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.708504 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.708377    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhlb\" (UniqueName: \"kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:36.727218 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.709260    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:36.892634 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:36.892604    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:37.056943 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.056850    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:37.136082 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.136035    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:37.138818 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.138791    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\") pod \"maas-api-e2e-derive-aa500779-84fddcd574-9n6sd\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:37.370133 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.370034    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:37.650699 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.650663    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 17:59:37.855648 ip-10-0-139-156 kubenswrapper[5838]: W1103 17:59:37.855604    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321f7f0e_5d10_478a_9173_531b70d1c4db.slice/crio-8fd52fb345e7fd1b44f35beb506a8c752105a9bf29ddd463967e0cfb856f08c7 WatchSource:0}: Error finding container 8fd52fb345e7fd1b44f35beb506a8c752105a9bf29ddd463967e0cfb856f08c7: Status 404 returned error can't find the container with id 8fd52fb345e7fd1b44f35beb506a8c752105a9bf29ddd463967e0cfb856f08c7
Nov 03 17:59:37.857048 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.857021    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"]
Nov 03 17:59:37.930168 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:37.930073    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" event={"ID":"321f7f0e-5d10-478a-9173-531b70d1c4db","Type":"ContainerStarted","Data":"8fd52fb345e7fd1b44f35beb506a8c752105a9bf29ddd463967e0cfb856f08c7"}
Nov 03 17:59:39.145939 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:39.145712    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:39.370670 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:39.370570    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:40.040789 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.037312    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" event={"ID":"321f7f0e-5d10-478a-9173-531b70d1c4db","Type":"ContainerStarted","Data":"ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c"}
Nov 03 17:59:40.040789 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.038669    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:40.233767 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.233203    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" podStartSLOduration=3.3282048570000002 podStartE2EDuration="4.233177605s" totalImagesPullingTime="904.972748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:40.169460922 +0000 UTC m=+1516.180198995" watchObservedRunningTime="2026-11-03 17:59:40.233177605 +0000 UTC m=+1516.243915678"
Nov 03 17:59:40.461895 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.461838    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:40.580057 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.580012    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:40.580666 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.580636    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:59:40.606461 ip-10-0-139-156 kubenswrapper[5838]: E1103 17:59:40.606419    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 17:59:40.656759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.631624    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:59:40.656759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.631716    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:40.983759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:40.978689    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:41.221668 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.221626    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 03 17:59:41.528759 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.505623    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:41.532515 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.532477    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:59:41.582940 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.582881    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:41.585255 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.585227    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:59:41.782322 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.782233    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 17:59:41.831343 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:41.808948    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 17:59:44.023805 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:44.021492    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:47.448647 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:47.448613    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 17:59:48.238982 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:48.238934    5838 patch_prober.go:29] interesting pod/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:59:48.239232 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:48.239034    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:59:49.350133 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:49.350090    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:50.533333 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:50.533240    5838 generic.go:356] "Generic (PLEG): container finished" podID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerID="df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96" exitCode=0
Nov 03 17:59:50.533333 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:50.533310    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" event={"ID":"d8babccd-6177-41a8-91c3-6cb2b4d52032","Type":"ContainerDied","Data":"df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"}
Nov 03 17:59:50.533922 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:50.533902    5838 scope.go:122] "RemoveContainer" containerID="df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"
Nov 03 17:59:50.535883 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:50.535849    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:50.535988 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:50.535927    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:50.535988 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:50.535963    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:51.545246 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:51.545203    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" event={"ID":"d8babccd-6177-41a8-91c3-6cb2b4d52032","Type":"ContainerStarted","Data":"9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837"}
Nov 03 17:59:52.165343 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:52.165303    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:59:52.165581 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:52.165558    5838 patch_prober.go:29] interesting pod/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 17:59:52.165651 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:52.165624    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 17:59:53.166158 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:53.166120    5838 patch_prober.go:29] interesting pod/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 17:59:53.166657 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:53.166176    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 17:59:54.165929 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:54.165884    5838 patch_prober.go:29] interesting pod/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 17:59:54.166132 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:54.165951    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 17:59:54.436786 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:54.436665    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:55.165879 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:55.165834    5838 patch_prober.go:29] interesting pod/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 17:59:55.166080 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:55.165902    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 17:59:56.165372 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:56.165330    5838 patch_prober.go:29] interesting pod/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 17:59:56.165803 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:56.165393    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 17:59:57.170289 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:57.170253    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:59:57.170722 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:57.170654    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:59:57.171417 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:57.171399    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 17:59:59.363212 ip-10-0-139-156 kubenswrapper[5838]: I1103 17:59:59.363175    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:00.211102 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.211064    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"]
Nov 03 18:00:00.215112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.215086    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.226362 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.226329    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-w77ts\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.226605 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.226586    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7n5\" (UniqueName: \"kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5\") pod \"maas-api-key-cleanup-29895480-w77ts\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.300560 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.300527    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:00.305845 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.305807    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"]
Nov 03 18:00:00.329316 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.329270    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7n5\" (UniqueName: \"kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5\") pod \"maas-api-key-cleanup-29895480-w77ts\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.329533 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.329514    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-w77ts\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.330239 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.330217    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-w77ts\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.331235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.331219    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:00:00.393645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.393608    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7n5\" (UniqueName: \"kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5\") pod \"maas-api-key-cleanup-29895480-w77ts\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.530928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.530842    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:00.920112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:00.919999    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"]
Nov 03 18:00:00.923683 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:00:00.923654    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded636ad5_8b41_4c1f_bf0e_2f9d34566a71.slice/crio-666a779bd99fcca22ca399fb89ce6693f25d2f9f3ab26c319344dbd8f6d0c3f2 WatchSource:0}: Error finding container 666a779bd99fcca22ca399fb89ce6693f25d2f9f3ab26c319344dbd8f6d0c3f2: Status 404 returned error can't find the container with id 666a779bd99fcca22ca399fb89ce6693f25d2f9f3ab26c319344dbd8f6d0c3f2
Nov 03 18:00:01.646251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:01.646201    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerStarted","Data":"666a779bd99fcca22ca399fb89ce6693f25d2f9f3ab26c319344dbd8f6d0c3f2"}
Nov 03 18:00:03.731616 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:03.731533    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:03.909847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:03.909802    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:05.137130 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:05.128333    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:10.358397 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:10.358344    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:00:10.595756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.595699    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"]
Nov 03 18:00:10.603698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.603665    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.658817 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.658786    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-12a446f9-serving-cert\""
Nov 03 18:00:10.714680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.714043    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"]
Nov 03 18:00:10.721964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.721923    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnk6b\" (UniqueName: \"kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.722191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.722077    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.722328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.722253    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.722328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.722295    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.790990 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.790958    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:10.823971 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.823933    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.824178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.823985    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.824178 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:10.824123    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-serving-cert: secret "maas-api-e2e-mt-12a446f9-serving-cert" not found
Nov 03 18:00:10.824178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.824152    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnk6b\" (UniqueName: \"kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.824352 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:10.824188    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls podName:1fa1a70f-2c2d-46fb-9e29-7d792c3dc274 nodeName:}" failed. No retries permitted until 2026-11-03 18:00:11.324171016 +0000 UTC m=+1547.334909067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls") pod "maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" (UID: "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274") : secret "maas-api-e2e-mt-12a446f9-serving-cert" not found
Nov 03 18:00:10.824352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.824307    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.825047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.825025    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.827071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.827042    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.890976 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.890903    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnk6b\" (UniqueName: \"kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:10.958892 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:10.958857    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:11.119940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:11.119892    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:11.335746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:11.335648    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:11.338307 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:11.338275    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\") pod \"maas-api-e2e-mt-12a446f9-78f9bb567f-s845k\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:11.541592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:11.541554    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:11.932923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:11.932889    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"]
Nov 03 18:00:11.933067 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:00:11.932954    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa1a70f_2c2d_46fb_9e29_7d792c3dc274.slice/crio-bbcba261ecb63abaa7fdc5b6e12a025c86f3c25c075fc961edda0c44c309d076 WatchSource:0}: Error finding container bbcba261ecb63abaa7fdc5b6e12a025c86f3c25c075fc961edda0c44c309d076: Status 404 returned error can't find the container with id bbcba261ecb63abaa7fdc5b6e12a025c86f3c25c075fc961edda0c44c309d076
Nov 03 18:00:12.754504 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:12.754404    5838 generic.go:356] "Generic (PLEG): container finished" podID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerID="ff6d9ea9d2268e11eaa8cb8871f1725f52ca22d1c00836bbcdb3d0f973d84f7f" exitCode=60
Nov 03 18:00:12.754989 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:12.754493    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerDied","Data":"ff6d9ea9d2268e11eaa8cb8871f1725f52ca22d1c00836bbcdb3d0f973d84f7f"}
Nov 03 18:00:12.754989 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:12.754907    5838 scope.go:122] "RemoveContainer" containerID="ff6d9ea9d2268e11eaa8cb8871f1725f52ca22d1c00836bbcdb3d0f973d84f7f"
Nov 03 18:00:12.756239 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:12.756218    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" event={"ID":"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274","Type":"ContainerStarted","Data":"bbcba261ecb63abaa7fdc5b6e12a025c86f3c25c075fc961edda0c44c309d076"}
Nov 03 18:00:13.766016 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:13.765978    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" event={"ID":"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274","Type":"ContainerStarted","Data":"47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071"}
Nov 03 18:00:13.766524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:13.766076    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:13.768760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:13.768717    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerStarted","Data":"b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1"}
Nov 03 18:00:13.892676 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:13.892618    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" podStartSLOduration=2.96236376 podStartE2EDuration="3.892598782s" totalImagesPullingTime="930.235022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:13.888662524 +0000 UTC m=+1549.899400615" watchObservedRunningTime="2026-11-03 18:00:13.892598782 +0000 UTC m=+1549.903336854"
Nov 03 18:00:14.010170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:14.010105    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" podStartSLOduration=3.582880565 podStartE2EDuration="14.010084386s" totalImagesPullingTime="10.427203821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:14.007209196 +0000 UTC m=+1550.017947294" watchObservedRunningTime="2026-11-03 18:00:14.010084386 +0000 UTC m=+1550.020822457"
Nov 03 18:00:14.649955 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:14.649916    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:14.780962 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:14.780929    5838 generic.go:356] "Generic (PLEG): container finished" podID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerID="b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1" exitCode=60
Nov 03 18:00:14.781444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:14.781017    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerDied","Data":"b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1"}
Nov 03 18:00:14.781444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:14.781109    5838 scope.go:122] "RemoveContainer" containerID="ff6d9ea9d2268e11eaa8cb8871f1725f52ca22d1c00836bbcdb3d0f973d84f7f"
Nov 03 18:00:14.781547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:14.781456    5838 scope.go:122] "RemoveContainer" containerID="b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1"
Nov 03 18:00:14.781939 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:14.781906    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-w77ts_odh-ai-gateway-infra(ed636ad5-8b41-4c1f-bf0e-2f9d34566a71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71"
Nov 03 18:00:15.339424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:15.339394    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:15.519902 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:15.519864    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:15.791592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:15.791560    5838 scope.go:122] "RemoveContainer" containerID="b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1"
Nov 03 18:00:15.792131 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:15.792018    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-w77ts_odh-ai-gateway-infra(ed636ad5-8b41-4c1f-bf0e-2f9d34566a71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71"
Nov 03 18:00:16.841964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:16.841921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:19.787006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:19.786975    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:00:22.390635 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:22.390599    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:27.109508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:27.109468    5838 scope.go:122] "RemoveContainer" containerID="b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1"
Nov 03 18:00:27.498908 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:27.498875    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:27.866188 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:27.866098    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:28.015591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:28.015547    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerStarted","Data":"f7b52b3d56d9f4796e7d8945f2d41afe60bb3653460d7c68e8a6c3816efc244f"}
Nov 03 18:00:28.028232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:28.028194    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:29.112294 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:29.112256    5838 generic.go:356] "Generic (PLEG): container finished" podID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerID="f7b52b3d56d9f4796e7d8945f2d41afe60bb3653460d7c68e8a6c3816efc244f" exitCode=60
Nov 03 18:00:29.113306 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:29.112953    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerDied","Data":"f7b52b3d56d9f4796e7d8945f2d41afe60bb3653460d7c68e8a6c3816efc244f"}
Nov 03 18:00:29.113514 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:29.113500    5838 scope.go:122] "RemoveContainer" containerID="b42988d7720573bcda7e7eb93989c00b72944928fab0fba8c55fc5bb628fd2c1"
Nov 03 18:00:29.118305 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:29.118280    5838 scope.go:122] "RemoveContainer" containerID="f7b52b3d56d9f4796e7d8945f2d41afe60bb3653460d7c68e8a6c3816efc244f"
Nov 03 18:00:29.119007 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:29.118961    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-w77ts_odh-ai-gateway-infra(ed636ad5-8b41-4c1f-bf0e-2f9d34566a71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71"
Nov 03 18:00:29.170658 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:29.170623    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"]
Nov 03 18:00:30.324563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.324533    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:30.402177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.402140    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca\") pod \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") "
Nov 03 18:00:30.402346 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.402259    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5\" (UniqueName: \"kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5\") pod \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\" (UID: \"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71\") "
Nov 03 18:00:30.402553 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.402527    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca" pod "ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" (UID: "ed636ad5-8b41-4c1f-bf0e-2f9d34566a71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:30.402921 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.402905    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-maas-api-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:30.404851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.404780    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5" pod "ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" (UID: "ed636ad5-8b41-4c1f-bf0e-2f9d34566a71"). InnerVolumeSpecName "kube-api-access-5z7n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:30.504365 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:30.504323    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7n5\" (UniqueName: \"kubernetes.io/projected/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71-kube-api-access-5z7n5\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:31.135614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:31.135571    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts" event={"ID":"ed636ad5-8b41-4c1f-bf0e-2f9d34566a71","Type":"ContainerDied","Data":"666a779bd99fcca22ca399fb89ce6693f25d2f9f3ab26c319344dbd8f6d0c3f2"}
Nov 03 18:00:31.135870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:31.135633    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"
Nov 03 18:00:31.135870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:31.135654    5838 scope.go:122] "RemoveContainer" containerID="f7b52b3d56d9f4796e7d8945f2d41afe60bb3653460d7c68e8a6c3816efc244f"
Nov 03 18:00:31.309532 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:31.309493    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"]
Nov 03 18:00:31.360219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:31.360180    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-w77ts"]
Nov 03 18:00:32.119691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:32.119655    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" path="/var/lib/kubelet/pods/ed636ad5-8b41-4c1f-bf0e-2f9d34566a71/volumes"
Nov 03 18:00:33.081500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:33.081471    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:38.503454 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:38.503415    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:39.510979 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:39.510934    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:40.343343 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:00:40.343298    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:00:43.630628 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:43.630585    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:49.501700 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:49.501657    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:54.969047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:54.969003    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:59.519153 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:00:59.519112    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:00.334198 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:00.334154    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:00.499371 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:00.499283    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:01.117199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:01.117161    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:01.387149 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:01.387066    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:01.558102 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:01.558065    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:01.922237 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:01.922201    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:01.943205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:01.943162    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"]
Nov 03 18:01:01.943462 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:01.943437    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy" containerID="cri-o://9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837" gracePeriod=30
Nov 03 18:01:02.612379 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:02.612346    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:03.300853 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:03.300815    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:03.586983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:03.586887    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:06.577097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:06.577064    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"]
Nov 03 18:01:06.577558 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:06.577279    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" podUID="ae26ee52-f1bb-4b72-b15d-a7f84274ab55" containerName="maas-api" containerID="cri-o://449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da" gracePeriod=30
Nov 03 18:01:06.675897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:06.675857    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:07.055613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.055577    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 18:01:07.201818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.187344    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\") pod \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") "
Nov 03 18:01:07.201818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.187422    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs\") pod \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") "
Nov 03 18:01:07.201818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.187457    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca\") pod \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") "
Nov 03 18:01:07.201818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.187498    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh\") pod \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\" (UID: \"ae26ee52-f1bb-4b72-b15d-a7f84274ab55\") "
Nov 03 18:01:07.214349 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.214289    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca" pod "ae26ee52-f1bb-4b72-b15d-a7f84274ab55" (UID: "ae26ee52-f1bb-4b72-b15d-a7f84274ab55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:07.225140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.216254    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs" pod "ae26ee52-f1bb-4b72-b15d-a7f84274ab55" (UID: "ae26ee52-f1bb-4b72-b15d-a7f84274ab55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:07.247033 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.246941    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls" pod "ae26ee52-f1bb-4b72-b15d-a7f84274ab55" (UID: "ae26ee52-f1bb-4b72-b15d-a7f84274ab55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:07.249425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.249368    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh" pod "ae26ee52-f1bb-4b72-b15d-a7f84274ab55" (UID: "ae26ee52-f1bb-4b72-b15d-a7f84274ab55"). InnerVolumeSpecName "kube-api-access-728sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:07.276955 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.276924    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 18:01:07.293573 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.293539    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.294809 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.294773    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.295903 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.295877    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.296088 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.296070    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/ae26ee52-f1bb-4b72-b15d-a7f84274ab55-kube-api-access-728sh\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.396961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.396931    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.396973    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397012    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397123    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397144    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397196    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397257    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:01:07.397441 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397273    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397441 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397426    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:07.397564 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397430    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397564 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397501    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket\") pod \"d8babccd-6177-41a8-91c3-6cb2b4d52032\" (UID: \"d8babccd-6177-41a8-91c3-6cb2b4d52032\") "
Nov 03 18:01:07.397846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397807    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:01:07.397969 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.397838    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:01:07.398219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.398196    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:01:07.398566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.398465    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.398566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.398492    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8babccd-6177-41a8-91c3-6cb2b4d52032-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.398566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.398508    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.398566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.398523    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.398566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.398539    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.399897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.399870    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:07.399897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.399883    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:01:07.400246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.400218    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:01:07.400343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.400321    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj" pod "d8babccd-6177-41a8-91c3-6cb2b4d52032" (UID: "d8babccd-6177-41a8-91c3-6cb2b4d52032"). InnerVolumeSpecName "kube-api-access-q7knj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:07.475183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.475038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:07.499764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.499718    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.499764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.499762    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.500019 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.499774    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.500019 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.499788    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7knj\" (UniqueName: \"kubernetes.io/projected/d8babccd-6177-41a8-91c3-6cb2b4d52032-kube-api-access-q7knj\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:07.570859 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.570825    5838 generic.go:356] "Generic (PLEG): container finished" podID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerID="9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837" exitCode=0
Nov 03 18:01:07.571063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.570893    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"
Nov 03 18:01:07.571063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.570917    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" event={"ID":"d8babccd-6177-41a8-91c3-6cb2b4d52032","Type":"ContainerDied","Data":"9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837"}
Nov 03 18:01:07.571063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.570982    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q" event={"ID":"d8babccd-6177-41a8-91c3-6cb2b4d52032","Type":"ContainerDied","Data":"587c55d51beccf3fb27da61cfb6f50edf01c3b0983ded112b4fe438b20c4f197"}
Nov 03 18:01:07.571063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.571010    5838 scope.go:122] "RemoveContainer" containerID="9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837"
Nov 03 18:01:07.573169 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.573147    5838 generic.go:356] "Generic (PLEG): container finished" podID="ae26ee52-f1bb-4b72-b15d-a7f84274ab55" containerID="449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da" exitCode=0
Nov 03 18:01:07.573278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.573192    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"
Nov 03 18:01:07.573278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.573207    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" event={"ID":"ae26ee52-f1bb-4b72-b15d-a7f84274ab55","Type":"ContainerDied","Data":"449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da"}
Nov 03 18:01:07.573278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.573247    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6" event={"ID":"ae26ee52-f1bb-4b72-b15d-a7f84274ab55","Type":"ContainerDied","Data":"5e7f7ec9781df6ce9361ba403495658d29ac0af712833e40b2eeceb50b85db83"}
Nov 03 18:01:07.587595 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.586906    5838 scope.go:122] "RemoveContainer" containerID="df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"
Nov 03 18:01:07.617604 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.617579    5838 scope.go:122] "RemoveContainer" containerID="9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837"
Nov 03 18:01:07.617967 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:01:07.617944    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837\": container with ID starting with 9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837 not found: ID does not exist" containerID="9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837"
Nov 03 18:01:07.618041 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.617977    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837"} err="failed to get container status \"9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837\": rpc error: code = NotFound desc = could not find container \"9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837\": container with ID starting with 9d158def94e0f7fa3fa255167479bc7644230ff116effd2b35ddc099d5eac837 not found: ID does not exist"
Nov 03 18:01:07.618041 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.617994    5838 scope.go:122] "RemoveContainer" containerID="df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"
Nov 03 18:01:07.618295 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:01:07.618275    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96\": container with ID starting with df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96 not found: ID does not exist" containerID="df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"
Nov 03 18:01:07.618355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.618304    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96"} err="failed to get container status \"df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96\": rpc error: code = NotFound desc = could not find container \"df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96\": container with ID starting with df2e92b4c129dff97597a53a61e8ab00c89f7c37c6c3eb271897f5b4a0969f96 not found: ID does not exist"
Nov 03 18:01:07.618355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.618321    5838 scope.go:122] "RemoveContainer" containerID="449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da"
Nov 03 18:01:07.629805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.629781    5838 scope.go:122] "RemoveContainer" containerID="449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da"
Nov 03 18:01:07.630131 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:01:07.630111    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da\": container with ID starting with 449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da not found: ID does not exist" containerID="449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da"
Nov 03 18:01:07.630239 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.630139    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da"} err="failed to get container status \"449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da\": rpc error: code = NotFound desc = could not find container \"449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da\": container with ID starting with 449bdd2dce71093971f236d4c0d1067d6f93f37cfd2196de97f31e07d7e3f6da not found: ID does not exist"
Nov 03 18:01:07.751151 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.751059    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:07.771316 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.771280    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"]
Nov 03 18:01:07.776154 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.776123    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c1ea78c-56b5f65c4-wmrt6"]
Nov 03 18:01:07.931759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.924153    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"]
Nov 03 18:01:07.966641 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:07.958036    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c74b2c26-gw-openshift-default-744dcbffc-q7v7q"]
Nov 03 18:01:08.028851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:08.028747    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:08.120085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:08.120052    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae26ee52-f1bb-4b72-b15d-a7f84274ab55" path="/var/lib/kubelet/pods/ae26ee52-f1bb-4b72-b15d-a7f84274ab55/volumes"
Nov 03 18:01:08.120617 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:08.120604    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" path="/var/lib/kubelet/pods/d8babccd-6177-41a8-91c3-6cb2b4d52032/volumes"
Nov 03 18:01:08.697252 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:08.697220    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:08.920298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:08.920258    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:09.131600 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:09.131448    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:09.457776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:09.457696    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:10.092490 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:10.092450    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:10.347851 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:01:10.346390    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:01:10.800755 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:10.793488    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:11.265034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:11.264987    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:11.451202 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:11.451172    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:11.694143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:11.694105    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:15.409254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:15.409202    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:16.120493 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:16.120452    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:17.044220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:17.044180    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:17.244271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:17.244181    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:17.423282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:17.423239    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:20.229761 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:20.225482    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:20.547233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:20.547138    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:01:21.155160 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:21.155127    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:21.318839 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:21.318801    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:21.655358 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:21.655309    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:21.905047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:21.905012    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:22.067875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:22.067774    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:27.138813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:27.138765    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:28.785221 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:28.785184    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:28.944953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:28.944921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:29.110020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:29.109929    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:32.714532 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:32.714337    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:36.627004 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:36.626967    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:37.798024 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:37.797978    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:39.853450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:39.853397    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:40.342917 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:01:40.342873    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:01:41.647207 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:41.647171    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:41.823080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:41.823048    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:42.225966 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.225933    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"]
Nov 03 18:01:42.226169 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.226147    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" podUID="321f7f0e-5d10-478a-9173-531b70d1c4db" containerName="maas-api" containerID="cri-o://ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c" gracePeriod=30
Nov 03 18:01:42.328135 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.328098    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:42.609898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.609863    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 18:01:42.727935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.727891    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb\" (UniqueName: \"kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb\") pod \"321f7f0e-5d10-478a-9173-531b70d1c4db\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") "
Nov 03 18:01:42.728439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.728018    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca\") pod \"321f7f0e-5d10-478a-9173-531b70d1c4db\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") "
Nov 03 18:01:42.728439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.728052    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs\") pod \"321f7f0e-5d10-478a-9173-531b70d1c4db\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") "
Nov 03 18:01:42.728439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.728211    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\") pod \"321f7f0e-5d10-478a-9173-531b70d1c4db\" (UID: \"321f7f0e-5d10-478a-9173-531b70d1c4db\") "
Nov 03 18:01:42.728439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.728423    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca" pod "321f7f0e-5d10-478a-9173-531b70d1c4db" (UID: "321f7f0e-5d10-478a-9173-531b70d1c4db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:42.729047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.729024    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f7f0e-5d10-478a-9173-531b70d1c4db-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:42.731127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.731024    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs" pod "321f7f0e-5d10-478a-9173-531b70d1c4db" (UID: "321f7f0e-5d10-478a-9173-531b70d1c4db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:42.731366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.731340    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb" pod "321f7f0e-5d10-478a-9173-531b70d1c4db" (UID: "321f7f0e-5d10-478a-9173-531b70d1c4db"). InnerVolumeSpecName "kube-api-access-zbhlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:42.732629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.732603    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls" pod "321f7f0e-5d10-478a-9173-531b70d1c4db" (UID: "321f7f0e-5d10-478a-9173-531b70d1c4db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:42.839256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.830843    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhlb\" (UniqueName: \"kubernetes.io/projected/321f7f0e-5d10-478a-9173-531b70d1c4db-kube-api-access-zbhlb\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:42.839256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.830883    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:42.839256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:42.830899    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321f7f0e-5d10-478a-9173-531b70d1c4db-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:43.196898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.196852    5838 generic.go:356] "Generic (PLEG): container finished" podID="321f7f0e-5d10-478a-9173-531b70d1c4db" containerID="ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c" exitCode=0
Nov 03 18:01:43.197166 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.197145    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" event={"ID":"321f7f0e-5d10-478a-9173-531b70d1c4db","Type":"ContainerDied","Data":"ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c"}
Nov 03 18:01:43.197235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.197203    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" event={"ID":"321f7f0e-5d10-478a-9173-531b70d1c4db","Type":"ContainerDied","Data":"8fd52fb345e7fd1b44f35beb506a8c752105a9bf29ddd463967e0cfb856f08c7"}
Nov 03 18:01:43.197341 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.197324    5838 scope.go:122] "RemoveContainer" containerID="ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c"
Nov 03 18:01:43.197439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.197324    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"
Nov 03 18:01:43.215227 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.215198    5838 scope.go:122] "RemoveContainer" containerID="ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c"
Nov 03 18:01:43.215624 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:01:43.215595    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c\": container with ID starting with ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c not found: ID does not exist" containerID="ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c"
Nov 03 18:01:43.215780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.215635    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c"} err="failed to get container status \"ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c\": rpc error: code = NotFound desc = could not find container \"ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c\": container with ID starting with ecc43ef54b0c328587104a6f13c8a75c67a2300b08b65d1a6a2044659fc9cb1c not found: ID does not exist"
Nov 03 18:01:43.375247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.375205    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"]
Nov 03 18:01:43.427496 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:43.427339    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd"]
Nov 03 18:01:44.119235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:44.119200    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321f7f0e-5d10-478a-9173-531b70d1c4db" path="/var/lib/kubelet/pods/321f7f0e-5d10-478a-9173-531b70d1c4db/volumes"
Nov 03 18:01:44.140193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:44.140155    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:44.417020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:44.416988    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:44.580599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:44.580565    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:44.809353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:44.809080    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:45.460524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:45.460482    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aa500779-84fddcd574-9n6sd" podUID="321f7f0e-5d10-478a-9173-531b70d1c4db" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.99:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:01:45.543705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:45.543672    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:46.217605 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:46.217561    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:46.385922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:46.385884    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:46.561077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:46.560983    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:47.936220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:47.936171    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:48.134205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:48.134163    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:48.306865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:48.306779    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:50.207402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:50.207365    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:55.592630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:01:55.592586    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:00.548831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.548784    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"]
Nov 03 18:02:00.550040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550012    5838 cpu_manager.go:414] "containerMap: removing container" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy"
Nov 03 18:02:00.550141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550041    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy"
Nov 03 18:02:00.550141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550077    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550086    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550109    5838 cpu_manager.go:414] "containerMap: removing container" podUID="321f7f0e-5d10-478a-9173-531b70d1c4db" containerName="maas-api"
Nov 03 18:02:00.550141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550118    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321f7f0e-5d10-478a-9173-531b70d1c4db" containerName="maas-api"
Nov 03 18:02:00.550308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550151    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550159    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550175    5838 cpu_manager.go:414] "containerMap: removing container" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy"
Nov 03 18:02:00.550308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550182    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy"
Nov 03 18:02:00.550308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550198    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ae26ee52-f1bb-4b72-b15d-a7f84274ab55" containerName="maas-api"
Nov 03 18:02:00.550308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550205    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae26ee52-f1bb-4b72-b15d-a7f84274ab55" containerName="maas-api"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550387    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550403    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="321f7f0e-5d10-478a-9173-531b70d1c4db" containerName="maas-api"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550415    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550425    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550435    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae26ee52-f1bb-4b72-b15d-a7f84274ab55" containerName="maas-api"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550449    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:00.550484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.550464    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8babccd-6177-41a8-91c3-6cb2b4d52032" containerName="istio-proxy"
Nov 03 18:02:00.556658 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.556629    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.617496 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.617461    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"]
Nov 03 18:02:00.663802 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.663765    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3e138c04-serving-cert\""
Nov 03 18:02:00.714199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.714157    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.714683 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.714655    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zvk\" (UniqueName: \"kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.714944 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.714924    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.715158 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.715141    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.749391 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.749335    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:00.816994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.816876    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.817189 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.817127    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zvk\" (UniqueName: \"kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.817189 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.817162    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.817317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.817233    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.820977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.818241    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.825456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.825429    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.825626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.825602    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.881499 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.881462    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zvk\" (UniqueName: \"kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk\") pod \"maas-api-e2e-ait-3e138c04-866c6db464-twlm6\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:00.927608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:00.927564    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:01.093835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:01.092566    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:01.176311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:01.176282    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:01.479062 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:01.479018    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:01.591161 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:01.589507    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"]
Nov 03 18:02:02.389282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:02.389235    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" event={"ID":"65e09abe-6887-40b5-a7c6-59c736652c25","Type":"ContainerStarted","Data":"653c02526a7593fd5fd8131e874b43f61fed3df18f5c95b5dde6217c2e2a1f93"}
Nov 03 18:02:03.399320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:03.399276    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" event={"ID":"65e09abe-6887-40b5-a7c6-59c736652c25","Type":"ContainerStarted","Data":"510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914"}
Nov 03 18:02:03.399827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:03.399342    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:03.524937 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:03.524883    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" podStartSLOduration=2.572201894 podStartE2EDuration="3.524867782s" totalImagesPullingTime="952.665888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:03.522806178 +0000 UTC m=+1659.533544252" watchObservedRunningTime="2026-11-03 18:02:03.524867782 +0000 UTC m=+1659.535605854"
Nov 03 18:02:05.515978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:05.515938    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"]
Nov 03 18:02:05.516476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:05.516188    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" podUID="1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" containerName="maas-api" containerID="cri-o://47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071" gracePeriod=30
Nov 03 18:02:05.633613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:05.633584    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:05.880994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:05.880965    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:02:06.019756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.019416    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs\") pod \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") "
Nov 03 18:02:06.019756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.019485    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b\" (UniqueName: \"kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b\") pod \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") "
Nov 03 18:02:06.019756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.019647    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\") pod \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") "
Nov 03 18:02:06.019756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.019751    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca\") pod \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\" (UID: \"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274\") "
Nov 03 18:02:06.020927 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.020898    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca" pod "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" (UID: "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:02:06.030893 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.028099    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b" pod "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" (UID: "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274"). InnerVolumeSpecName "kube-api-access-xnk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:02:06.036991 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.031339    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls" pod "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" (UID: "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:02:06.037705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.037669    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs" pod "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" (UID: "1fa1a70f-2c2d-46fb-9e29-7d792c3dc274"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:02:06.129155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.129005    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:02:06.129155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.129043    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnk6b\" (UniqueName: \"kubernetes.io/projected/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-kube-api-access-xnk6b\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:02:06.129155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.129059    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:02:06.129155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.129073    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:02:06.425186 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.425148    5838 generic.go:356] "Generic (PLEG): container finished" podID="1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" containerID="47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071" exitCode=0
Nov 03 18:02:06.425394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.425210    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"
Nov 03 18:02:06.425394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.425238    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" event={"ID":"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274","Type":"ContainerDied","Data":"47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071"}
Nov 03 18:02:06.425394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.425292    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k" event={"ID":"1fa1a70f-2c2d-46fb-9e29-7d792c3dc274","Type":"ContainerDied","Data":"bbcba261ecb63abaa7fdc5b6e12a025c86f3c25c075fc961edda0c44c309d076"}
Nov 03 18:02:06.425394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.425326    5838 scope.go:122] "RemoveContainer" containerID="47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071"
Nov 03 18:02:06.439465 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.439431    5838 scope.go:122] "RemoveContainer" containerID="47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071"
Nov 03 18:02:06.439849 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:02:06.439824    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071\": container with ID starting with 47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071 not found: ID does not exist" containerID="47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071"
Nov 03 18:02:06.439913 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.439858    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071"} err="failed to get container status \"47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071\": rpc error: code = NotFound desc = could not find container \"47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071\": container with ID starting with 47b0814c927cf12d006057493e041ce3e98fa9d4b6b81ac07796dc229cc06071 not found: ID does not exist"
Nov 03 18:02:06.609418 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.609340    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"]
Nov 03 18:02:06.616245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:06.616208    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12a446f9-78f9bb567f-s845k"]
Nov 03 18:02:07.336401 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.336365    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"]
Nov 03 18:02:07.337146 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.337131    5838 cpu_manager.go:414] "containerMap: removing container" podUID="1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" containerName="maas-api"
Nov 03 18:02:07.337206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.337147    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" containerName="maas-api"
Nov 03 18:02:07.337206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.337195    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:07.337206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.337201    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed636ad5-8b41-4c1f-bf0e-2f9d34566a71" containerName="cleanup"
Nov 03 18:02:07.337325 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.337309    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" containerName="maas-api"
Nov 03 18:02:07.342353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.342330    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.403394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.403364    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-394741e0-serving-cert\""
Nov 03 18:02:07.404016 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.403987    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"]
Nov 03 18:02:07.413135 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.413090    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:07.460510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.460470    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.460696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.460619    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.460696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.460678    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdl7\" (UniqueName: \"kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.460925 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.460869    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.562251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.562206    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.562571 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.562304    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.562571 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.562353    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdl7\" (UniqueName: \"kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.562571 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.562454    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.562723 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:02:07.562626    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-serving-cert: secret "maas-api-e2e-ait-394741e0-serving-cert" not found
Nov 03 18:02:07.562723 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:02:07.562709    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls podName:a96d240a-7e75-4e08-aa91-5697565a38cb nodeName:}" failed. No retries permitted until 2026-11-03 18:02:08.062687431 +0000 UTC m=+1664.073425484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls") pod "maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" (UID: "a96d240a-7e75-4e08-aa91-5697565a38cb") : secret "maas-api-e2e-ait-394741e0-serving-cert" not found
Nov 03 18:02:07.563065 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.563047    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.565057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.565036    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.583769 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.583714    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:07.626810 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.626715    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdl7\" (UniqueName: \"kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:07.742197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:07.742167    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:08.072916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:08.072873    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:08.075691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:08.075658    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\") pod \"maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:08.120644 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:08.120606    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa1a70f-2c2d-46fb-9e29-7d792c3dc274" path="/var/lib/kubelet/pods/1fa1a70f-2c2d-46fb-9e29-7d792c3dc274/volumes"
Nov 03 18:02:08.258414 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:08.258371    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:08.430831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:08.430783    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:08.652437 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:08.652394    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"]
Nov 03 18:02:08.653067 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:02:08.653045    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96d240a_7e75_4e08_aa91_5697565a38cb.slice/crio-81736369ef9196d72b6ec8e1c8dbb61662fb198b24b49dea0b305fcc04f307c6 WatchSource:0}: Error finding container 81736369ef9196d72b6ec8e1c8dbb61662fb198b24b49dea0b305fcc04f307c6: Status 404 returned error can't find the container with id 81736369ef9196d72b6ec8e1c8dbb61662fb198b24b49dea0b305fcc04f307c6
Nov 03 18:02:09.414585 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:09.414554    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:02:09.469886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:09.469857    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" event={"ID":"a96d240a-7e75-4e08-aa91-5697565a38cb","Type":"ContainerStarted","Data":"81736369ef9196d72b6ec8e1c8dbb61662fb198b24b49dea0b305fcc04f307c6"}
Nov 03 18:02:09.692797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:09.692666    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:10.345049 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:02:10.344999    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:02:10.479596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:10.479551    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" event={"ID":"a96d240a-7e75-4e08-aa91-5697565a38cb","Type":"ContainerStarted","Data":"9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c"}
Nov 03 18:02:10.479822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:10.479628    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:10.606277 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:10.606167    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" podStartSLOduration=2.812081057 podStartE2EDuration="3.606152344s" totalImagesPullingTime="794.071287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:10.603422282 +0000 UTC m=+1666.614160355" watchObservedRunningTime="2026-11-03 18:02:10.606152344 +0000 UTC m=+1666.616890417"
Nov 03 18:02:11.624964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.624929    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"]
Nov 03 18:02:11.630040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.630014    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660010 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.659977    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660188 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660025    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qgv\" (UniqueName: \"kubernetes.io/projected/aaff885c-bbb0-43e2-92f7-97a48d101d52-kube-api-access-l8qgv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660188 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660047    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660188 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660168    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660339 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660317    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660387 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660373    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaff885c-bbb0-43e2-92f7-97a48d101d52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660494 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660482    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660551 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660538    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.660649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.660634    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.729311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.729276    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"]
Nov 03 18:02:11.762171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762127    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762213    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaff885c-bbb0-43e2-92f7-97a48d101d52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762307    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762513 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762386    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762513 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762449    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762550    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762594    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qgv\" (UniqueName: \"kubernetes.io/projected/aaff885c-bbb0-43e2-92f7-97a48d101d52-kube-api-access-l8qgv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762621    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762817 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762692    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.762983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.762958    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.763247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.763222    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.763397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.763371    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.763591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.763570    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.763931 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.763882    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaff885c-bbb0-43e2-92f7-97a48d101d52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.766345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.766305    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.766988 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.766955    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.828312 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.828276    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaff885c-bbb0-43e2-92f7-97a48d101d52-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.828508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.828446    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qgv\" (UniqueName: \"kubernetes.io/projected/aaff885c-bbb0-43e2-92f7-97a48d101d52-kube-api-access-l8qgv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8jw2g\" (UID: \"aaff885c-bbb0-43e2-92f7-97a48d101d52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:11.951125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:11.951094    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:12.405513 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:02:12.405480    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaff885c_bbb0_43e2_92f7_97a48d101d52.slice/crio-75fdfe41616dc6a235c57855d3c0ca275cc01970639f37bc9b62803beacee3ae WatchSource:0}: Error finding container 75fdfe41616dc6a235c57855d3c0ca275cc01970639f37bc9b62803beacee3ae: Status 404 returned error can't find the container with id 75fdfe41616dc6a235c57855d3c0ca275cc01970639f37bc9b62803beacee3ae
Nov 03 18:02:12.406129 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.406105    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"]
Nov 03 18:02:12.408423 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.408382    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:12.408543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.408460    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:12.408543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.408500    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:12.506012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.505960    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" event={"ID":"aaff885c-bbb0-43e2-92f7-97a48d101d52","Type":"ContainerStarted","Data":"eb1e9a7356806dde0fc35bd2d437e69ff3f19ba67f972322d2a4c83cd9783e1a"}
Nov 03 18:02:12.506179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.506031    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" event={"ID":"aaff885c-bbb0-43e2-92f7-97a48d101d52","Type":"ContainerStarted","Data":"75fdfe41616dc6a235c57855d3c0ca275cc01970639f37bc9b62803beacee3ae"}
Nov 03 18:02:12.638058 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.637975    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podStartSLOduration=1.637947498 podStartE2EDuration="1.637947498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:12.632901944 +0000 UTC m=+1668.643640057" watchObservedRunningTime="2026-11-03 18:02:12.637947498 +0000 UTC m=+1668.648685572"
Nov 03 18:02:12.951723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:12.951685    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:13.230819 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:13.230411    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:13.954143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:13.954103    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:13.954554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:13.954182    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:14.247847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:14.247759    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:14.948796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:14.947112    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:14.973612 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:14.957681    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:14.973612 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:14.957777    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:15.360054 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:15.359960    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:15.951672 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:15.951637    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:15.951940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:15.951702    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:16.493048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:16.493018    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:02:16.595534 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:16.595494    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:16.759888 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:16.754078    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:16.959294 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:16.959256    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:16.960882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:16.960844    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:17.023832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:17.016637    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:17.544780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:17.544712    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:17.702631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:17.702580    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:17.952533 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:17.952497    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:17.952716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:17.952579    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:18.087396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:18.087345    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:18.635295 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:18.635238    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:18.952126 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:18.952084    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:18.952351 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:18.952153    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:19.951993 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:19.951953    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 18:02:19.952443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:19.952021    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 18:02:19.973466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:19.973411    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:20.147555 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:20.147525    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:20.333796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:20.333681    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:21.327236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:21.327195    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:21.327707 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:21.327598    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:21.329220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:21.329196    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g"
Nov 03 18:02:23.278543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:23.278506    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:23.488827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:23.486297    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:25.003931 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:25.003801    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:25.713807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:25.713722    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:25.880492 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:25.880464    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:27.234121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:27.234089    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:29.340960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:29.340923    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:29.508254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:29.508223    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:30.353987 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:30.353944    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:34.310898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:34.310849    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:34.487338 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:34.487275    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:34.681827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:34.681795    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:35.640680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:35.640648    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:37.671458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:37.671404    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:37.841895 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:37.841856    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:40.326019 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:40.325966    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:40.345636 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:02:40.345593    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:02:40.484225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:40.484196    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:41.212193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:41.212150    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:46.298326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:46.298296    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:47.054035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:47.053998    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:02:50.696896 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:50.696850    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:02:50.697506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:50.697469    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:02:52.110108 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:52.110013    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:56.820146 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.820102    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"]
Nov 03 18:02:56.825240 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.825206    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.937793 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.937752    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"]
Nov 03 18:02:56.942689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.942657    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-394741e0-openshift-default-dockercfg-hxf6x\""
Nov 03 18:02:56.970789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.970725    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971005 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.970855    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971005 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.970916    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.971058    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.971195    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971279 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.971236    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971362 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.971346    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.971413    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wql\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:56.971545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:56.971528    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.072926 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.072840    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.072939    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.072977    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073021    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073074    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073176    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073201    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073384 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073294    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073384 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073324    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wql\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073384 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073357    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073537    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073657 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073599    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073657 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073614    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.073995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.073978    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.075513 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.075490    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.075702 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.075684    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.138073 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.138027    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.138423 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.138397    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wql\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.143328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.143305    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:57.588104 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:02:57.588064    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29a21d5_12e3_4bc2_9581_58ed132c851d.slice/crio-5eec105f3009686ece73936c295820e6c1f512019e45269fedb1a52b29305910 WatchSource:0}: Error finding container 5eec105f3009686ece73936c295820e6c1f512019e45269fedb1a52b29305910: Status 404 returned error can't find the container with id 5eec105f3009686ece73936c295820e6c1f512019e45269fedb1a52b29305910
Nov 03 18:02:57.588363 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.588341    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"]
Nov 03 18:02:57.590993 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.590953    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:57.591100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.591031    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:57.591100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.591062    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:57.794464 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:57.794197    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:58.140185 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:58.140143    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" event={"ID":"c29a21d5-12e3-4bc2-9581-58ed132c851d","Type":"ContainerStarted","Data":"cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557"}
Nov 03 18:02:58.140708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:58.140213    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" event={"ID":"c29a21d5-12e3-4bc2-9581-58ed132c851d","Type":"ContainerStarted","Data":"5eec105f3009686ece73936c295820e6c1f512019e45269fedb1a52b29305910"}
Nov 03 18:02:58.144139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:58.144110    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:02:58.145493 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:58.145466    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:02:58.145645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:58.145523    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 18:02:58.269301 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:58.269241    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podStartSLOduration=2.269226072 podStartE2EDuration="2.269226072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:58.26658953 +0000 UTC m=+1714.277327621" watchObservedRunningTime="2026-11-03 18:02:58.269226072 +0000 UTC m=+1714.279964143"
Nov 03 18:02:59.144065 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:59.144034    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:02:59.144511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:59.144084    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 18:02:59.336922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:59.336887    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:59.529627 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:02:59.529506    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:00.143956 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:00.143921    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:03:00.144126 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:00.143986    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 18:03:00.339661 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:00.339596    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:01.144612 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:01.144579    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:03:01.145006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:01.144637    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 18:03:02.143952 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:02.143910    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:03:02.144177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:02.143967    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 18:03:03.143908 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:03.143875    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:03:03.144420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:03.143939    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 18:03:03.267569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:03.267534    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:04.148798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:04.148768    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:03:04.149275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:04.149222    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:03:04.150107 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:04.150084    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:03:09.286892 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:09.286858    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:10.342875 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:10.342819    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:03:11.584282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:11.584248    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"]
Nov 03 18:03:11.584916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:11.584479    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" containerID="cri-o://06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c" gracePeriod=30
Nov 03 18:03:11.869941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:11.869858    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"]
Nov 03 18:03:11.874910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:11.874882    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:11.960940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:11.960910    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"]
Nov 03 18:03:12.004828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.004784    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.005343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.005319    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.005576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.005554    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.005718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.005665    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcrf\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.005855 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.005809    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.005924 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.005858    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.006041 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.006022    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.006268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.006245    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.006374 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.006290    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.108101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.108065    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.108320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.108140    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.108320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.108282    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.108608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.108580    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.108954 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.108928    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.109273    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.109354    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.109377    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.109443    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.109574    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcrf\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.110058    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.110412    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.110995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.110514    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.111994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.111889    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.111994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.111951    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.114142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.114005    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.124863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.124796    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"]
Nov 03 18:03:12.131217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.131197    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.176935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.176907    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.177201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.177176    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcrf\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.190860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.190828    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:12.211806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.211562    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.211806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.211662    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.214866 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.214774    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.215110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.215025    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.217743 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.216881    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc862fa-db10-4720-9d9b-90d4533a216c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.217743 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.217019    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc862fa-db10-4720-9d9b-90d4533a216c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.217743 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.217115    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gt2l\" (UniqueName: \"kubernetes.io/projected/efc862fa-db10-4720-9d9b-90d4533a216c-kube-api-access-5gt2l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.217743 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.217247    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.218393 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.218365    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc862fa-db10-4720-9d9b-90d4533a216c-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.227722 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.227680    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"]
Nov 03 18:03:12.321353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321139    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321191    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321257    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321293    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc862fa-db10-4720-9d9b-90d4533a216c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321722 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321383    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc862fa-db10-4720-9d9b-90d4533a216c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321722 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321439    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gt2l\" (UniqueName: \"kubernetes.io/projected/efc862fa-db10-4720-9d9b-90d4533a216c-kube-api-access-5gt2l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321722 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321548    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.321722 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321705    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc862fa-db10-4720-9d9b-90d4533a216c-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.322055 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.321897    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.323077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.322995    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc862fa-db10-4720-9d9b-90d4533a216c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.323817 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.323787    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.324343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.324315    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.324610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.324585    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.324873 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.324848    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.327176 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.327148    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc862fa-db10-4720-9d9b-90d4533a216c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.328950 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.328919    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc862fa-db10-4720-9d9b-90d4533a216c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.386296 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.386170    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gt2l\" (UniqueName: \"kubernetes.io/projected/efc862fa-db10-4720-9d9b-90d4533a216c-kube-api-access-5gt2l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.388321 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.388294    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc862fa-db10-4720-9d9b-90d4533a216c-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-ml2hv\" (UID: \"efc862fa-db10-4720-9d9b-90d4533a216c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.467118 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.467086    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:12.668196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.668158    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"]
Nov 03 18:03:12.670119 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:03:12.670081    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb17c87_66d7_428e_a882_aa14eaaa984a.slice/crio-826b1459454aa87cc5076456cd75eba06aba1e02242cdb50680e9672fbb5e5da WatchSource:0}: Error finding container 826b1459454aa87cc5076456cd75eba06aba1e02242cdb50680e9672fbb5e5da: Status 404 returned error can't find the container with id 826b1459454aa87cc5076456cd75eba06aba1e02242cdb50680e9672fbb5e5da
Nov 03 18:03:12.676699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.676665    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.676805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.676760    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.676863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.676819    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.874227 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.874196    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"]
Nov 03 18:03:12.874784 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:03:12.874707    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc862fa_db10_4720_9d9b_90d4533a216c.slice/crio-8af1edb83b03aad30609b260777bb65bd6042551244f0aa82ea4136a1c666ec4 WatchSource:0}: Error finding container 8af1edb83b03aad30609b260777bb65bd6042551244f0aa82ea4136a1c666ec4: Status 404 returned error can't find the container with id 8af1edb83b03aad30609b260777bb65bd6042551244f0aa82ea4136a1c666ec4
Nov 03 18:03:12.878169 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.878130    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.878258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.878238    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.878315 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:12.878299    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:13.314595 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.313518    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" event={"ID":"5bb17c87-66d7-428e-a882-aa14eaaa984a","Type":"ContainerStarted","Data":"bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d"}
Nov 03 18:03:13.314595 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.313589    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" event={"ID":"5bb17c87-66d7-428e-a882-aa14eaaa984a","Type":"ContainerStarted","Data":"826b1459454aa87cc5076456cd75eba06aba1e02242cdb50680e9672fbb5e5da"}
Nov 03 18:03:13.315833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.315801    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" event={"ID":"efc862fa-db10-4720-9d9b-90d4533a216c","Type":"ContainerStarted","Data":"bacca45aca43f7029c1223d97713905578512315fae12c5205c6ebb3c2e152ae"}
Nov 03 18:03:13.315987 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.315861    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" event={"ID":"efc862fa-db10-4720-9d9b-90d4533a216c","Type":"ContainerStarted","Data":"8af1edb83b03aad30609b260777bb65bd6042551244f0aa82ea4136a1c666ec4"}
Nov 03 18:03:13.472872 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.472724    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:13.474475 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.474431    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:13.474791 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.474759    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:13.574792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.574678    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podStartSLOduration=2.574662912 podStartE2EDuration="2.574662912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:13.448334853 +0000 UTC m=+1729.459072922" watchObservedRunningTime="2026-11-03 18:03:13.574662912 +0000 UTC m=+1729.585400983"
Nov 03 18:03:13.575810 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:13.575761    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podStartSLOduration=1.57574883 podStartE2EDuration="1.57574883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:13.573516702 +0000 UTC m=+1729.584254798" watchObservedRunningTime="2026-11-03 18:03:13.57574883 +0000 UTC m=+1729.586486896"
Nov 03 18:03:14.193984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.193939    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:14.195300 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.195260    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.195448 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.195341    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 18:03:14.468637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.468497    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.468637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.468574    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:14.471406 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.471372    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"]
Nov 03 18:03:14.478207 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.478181    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.535757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.535331    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-e75517-serving-cert\""
Nov 03 18:03:14.577147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.576117    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcvr\" (UniqueName: \"kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.577147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.576351    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.577147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.576574    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.577147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.576836    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.579597 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.579570    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"]
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.682062    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.682276    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.682468    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcvr\" (UniqueName: \"kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.682566    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.683523    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:14.683610    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-serving-cert: secret "maas-api-e2e-shared-a-w0-e75517-serving-cert" not found
Nov 03 18:03:14.683782 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:14.683692    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls podName:cede57ba-951a-4f4b-a689-5cf9bfedc3dd nodeName:}" failed. No retries permitted until 2026-11-03 18:03:15.183669188 +0000 UTC m=+1731.194407242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls") pod "maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" (UID: "cede57ba-951a-4f4b-a689-5cf9bfedc3dd") : secret "maas-api-e2e-shared-a-w0-e75517-serving-cert" not found
Nov 03 18:03:14.684855 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.684541    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.688062 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.688044    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:14.883869 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:14.869214    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:15.035767 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.030405    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcvr\" (UniqueName: \"kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:15.188247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.188205    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:15.190110 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:15.189686    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-serving-cert: secret "maas-api-e2e-shared-a-w0-e75517-serving-cert" not found
Nov 03 18:03:15.190110 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:15.189792    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls podName:cede57ba-951a-4f4b-a689-5cf9bfedc3dd nodeName:}" failed. No retries permitted until 2026-11-03 18:03:16.189767982 +0000 UTC m=+1732.200506046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls") pod "maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" (UID: "cede57ba-951a-4f4b-a689-5cf9bfedc3dd") : secret "maas-api-e2e-shared-a-w0-e75517-serving-cert" not found
Nov 03 18:03:15.191770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.191497    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.191770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.191550    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 18:03:15.468356 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.468262    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.468820 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.468368    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:15.472177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:15.472139    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:16.192246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.192210    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.192447 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.192280    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 18:03:16.249433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.249389    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:16.264888 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.264852    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:16.313120 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.313081    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:16.360208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.359851    5838 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:03:16.360208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.359918    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:03:16.373553 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.373527    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:16.468506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.468296    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.468506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.468360    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:16.881757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.880590    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"]
Nov 03 18:03:16.886683 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:16.886370    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:03:17.192139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.192085    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.192480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.192419    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 18:03:17.435812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.435041    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 18:03:17.491760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.488584    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.491760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.488689    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:17.581963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.574189    5838 generic.go:356] "Generic (PLEG): container finished" podID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerID="06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c" exitCode=0
Nov 03 18:03:17.581963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.574374    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" event={"ID":"6cd5658a-c97e-44b2-9375-43983eb9eac1","Type":"ContainerDied","Data":"06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c"}
Nov 03 18:03:17.581963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.574484    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm" event={"ID":"6cd5658a-c97e-44b2-9375-43983eb9eac1","Type":"ContainerDied","Data":"e91a70915177e91e6f96584e8fc1911f611ee55a1e68fcbee6d4dde560b68f50"}
Nov 03 18:03:17.581963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.574517    5838 scope.go:122] "RemoveContainer" containerID="06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c"
Nov 03 18:03:17.581963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.574683    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"
Nov 03 18:03:17.582871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.582241    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" event={"ID":"cede57ba-951a-4f4b-a689-5cf9bfedc3dd","Type":"ContainerStarted","Data":"033cc1fb622b5aa234b5aafa5bfa53e0155ef769bdd34055dcb1e61e9df995d7"}
Nov 03 18:03:17.591787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.591714    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.591959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.591859    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592032 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.591999    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.592035    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.592070    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.592186    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.592217    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.592319    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.592455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.592438    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy\") pod \"6cd5658a-c97e-44b2-9375-43983eb9eac1\" (UID: \"6cd5658a-c97e-44b2-9375-43983eb9eac1\") "
Nov 03 18:03:17.595671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.593569    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:17.595671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.594025    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:17.595671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.594273    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:17.595671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.595519    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:17.595671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.595608    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:17.629318 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.628378    5838 scope.go:122] "RemoveContainer" containerID="06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c"
Nov 03 18:03:17.645827 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:17.629599    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c\": container with ID starting with 06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c not found: ID does not exist" containerID="06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c"
Nov 03 18:03:17.645827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.629648    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c"} err="failed to get container status \"06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c\": rpc error: code = NotFound desc = could not find container \"06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c\": container with ID starting with 06789650e1beef6c0ebacc7913c2a46ae32bbcf5f59bfbbd99f3d91ba268586c not found: ID does not exist"
Nov 03 18:03:17.709526 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.709491    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd5658a-c97e-44b2-9375-43983eb9eac1-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.709720 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.709706    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.709867 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.709852    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.709969 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.709958    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.710048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.710038    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.730433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.729935    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:17.737247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.737205    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "kube-api-access-tkhxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:17.795972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.794378    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:03:17.795972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.794627    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token" pod "6cd5658a-c97e-44b2-9375-43983eb9eac1" (UID: "6cd5658a-c97e-44b2-9375-43983eb9eac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:17.841209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.840955    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.878841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.878794    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhxv\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-kube-api-access-tkhxv\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.878841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.878841    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.879173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:17.878868    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd5658a-c97e-44b2-9375-43983eb9eac1-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:17.992458 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:17.992370    5838 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd5658a_c97e_44b2_9375_43983eb9eac1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 18:03:18.092396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.089769    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"]
Nov 03 18:03:18.127796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.117052    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-tcqrm"]
Nov 03 18:03:18.181077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.181041    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" path="/var/lib/kubelet/pods/6cd5658a-c97e-44b2-9375-43983eb9eac1/volumes"
Nov 03 18:03:18.192458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.192428    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:03:18.192662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.192639    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 18:03:18.475518 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.471003    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:18.475518 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.471063    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:18.661353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.658013    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:18.661353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.661164    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" event={"ID":"cede57ba-951a-4f4b-a689-5cf9bfedc3dd","Type":"ContainerStarted","Data":"fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11"}
Nov 03 18:03:18.666842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.663086    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:18.841516 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.833513    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" podStartSLOduration=3.619708425 podStartE2EDuration="4.833488613s" totalImagesPullingTime="1.213780188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:18.817056257 +0000 UTC m=+1734.827794330" watchObservedRunningTime="2026-11-03 18:03:18.833488613 +0000 UTC m=+1734.844226693"
Nov 03 18:03:18.893180 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:18.891017    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:19.197201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:19.197163    5838 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:03:19.197524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:19.197494    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 18:03:19.468431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:19.468268    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:19.468431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:19.468351    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:20.219770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:20.210018    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:20.219770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:20.211823    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:20.231759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:20.226384    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:03:20.467910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:20.467873    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:20.468106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:20.467933    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:21.467747 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:21.467691    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:03:21.468209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:21.467772    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 18:03:22.742013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:22.741959    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:22.891248 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:22.891214    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:22.892705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:22.892604    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:22.895213 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:22.895184    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv"
Nov 03 18:03:23.896390 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:23.896353    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:25.695618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:25.695570    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:03:26.720166 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:26.720131    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"]
Nov 03 18:03:26.720620 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:26.720351    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy" containerID="cri-o://61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34" gracePeriod=30
Nov 03 18:03:26.769792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:26.769724    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"]
Nov 03 18:03:26.770040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:26.770015    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy" containerID="cri-o://c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107" gracePeriod=30
Nov 03 18:03:26.950693 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:26.950624    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:28.415355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:28.415310    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:28.622209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:28.622179    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:28.781507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:28.781414    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:32.083765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.083716    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 18:03:32.126138 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.126103    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 18:03:32.213559 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213524    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.213797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213580    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.213797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213608    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.213797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213629    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.213797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213725    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.213797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213785    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213911    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213927    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.213990    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214021    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214066    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214076    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.214086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214081    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214261    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214279    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214296    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214359    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214416    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214475    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.214503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214484    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.215100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214506    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.215100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214547    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo\") pod \"916ebdff-e9a4-47b1-ab2a-779238cd7430\" (UID: \"916ebdff-e9a4-47b1-ab2a-779238cd7430\") "
Nov 03 18:03:32.215100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214596    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.215100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214678    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl\") pod \"ddace43b-4112-4594-92df-095167b2bb3f\" (UID: \"ddace43b-4112-4594-92df-095167b2bb3f\") "
Nov 03 18:03:32.215100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.214825    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.215100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215060    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.215835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215805    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215837    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddace43b-4112-4594-92df-095167b2bb3f-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215855    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215870    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916ebdff-e9a4-47b1-ab2a-779238cd7430-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215883    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215898    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215912    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.215949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.215925    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.216450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.216423    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.216600 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.216541    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.217015 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.216974    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "kube-api-access-vcwsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:32.217015 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.216974    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.217154 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.217086    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:32.217288 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.217265    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:32.217532 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.217512    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:32.217594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.217529    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "kube-api-access-x7ntl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:32.218118 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.218099    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo" pod "ddace43b-4112-4594-92df-095167b2bb3f" (UID: "ddace43b-4112-4594-92df-095167b2bb3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:03:32.218620 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.218599    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo" pod "916ebdff-e9a4-47b1-ab2a-779238cd7430" (UID: "916ebdff-e9a4-47b1-ab2a-779238cd7430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318056    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwsc\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-kube-api-access-vcwsc\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318095    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318111    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318124    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318138    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddace43b-4112-4594-92df-095167b2bb3f-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318153    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916ebdff-e9a4-47b1-ab2a-779238cd7430-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318167    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318180    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddace43b-4112-4594-92df-095167b2bb3f-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318195    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ntl\" (UniqueName: \"kubernetes.io/projected/ddace43b-4112-4594-92df-095167b2bb3f-kube-api-access-x7ntl\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.318842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.318207    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916ebdff-e9a4-47b1-ab2a-779238cd7430-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:32.829657 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.829614    5838 generic.go:356] "Generic (PLEG): container finished" podID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerID="c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107" exitCode=0
Nov 03 18:03:32.829890 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.829663    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"
Nov 03 18:03:32.829965 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.829659    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" event={"ID":"916ebdff-e9a4-47b1-ab2a-779238cd7430","Type":"ContainerDied","Data":"c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107"}
Nov 03 18:03:32.829965 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.829954    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp" event={"ID":"916ebdff-e9a4-47b1-ab2a-779238cd7430","Type":"ContainerDied","Data":"a7d4c3313043463f47471cc21f8b2a4277dd041612e58a95a655ab8cfdace28a"}
Nov 03 18:03:32.830080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.829988    5838 scope.go:122] "RemoveContainer" containerID="c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107"
Nov 03 18:03:32.831746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.831700    5838 generic.go:356] "Generic (PLEG): container finished" podID="ddace43b-4112-4594-92df-095167b2bb3f" containerID="61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34" exitCode=0
Nov 03 18:03:32.831970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.831793    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" event={"ID":"ddace43b-4112-4594-92df-095167b2bb3f","Type":"ContainerDied","Data":"61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34"}
Nov 03 18:03:32.831970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.831812    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"
Nov 03 18:03:32.831970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.831840    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj" event={"ID":"ddace43b-4112-4594-92df-095167b2bb3f","Type":"ContainerDied","Data":"8861b75c6b997a1ae4d2b93bcb9d8c8123969c9f794b726bee143c47d3cffda4"}
Nov 03 18:03:32.843173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.843152    5838 scope.go:122] "RemoveContainer" containerID="c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107"
Nov 03 18:03:32.843759 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:32.843580    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107\": container with ID starting with c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107 not found: ID does not exist" containerID="c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107"
Nov 03 18:03:32.843886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.843774    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107"} err="failed to get container status \"c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107\": rpc error: code = NotFound desc = could not find container \"c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107\": container with ID starting with c2ad23dabb28412b5fd4bbe53b4560ca477beaa923cd36643ce590ff05da9107 not found: ID does not exist"
Nov 03 18:03:32.843886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.843807    5838 scope.go:122] "RemoveContainer" containerID="61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34"
Nov 03 18:03:32.856289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.856270    5838 scope.go:122] "RemoveContainer" containerID="61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34"
Nov 03 18:03:32.856611 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:32.856584    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34\": container with ID starting with 61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34 not found: ID does not exist" containerID="61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34"
Nov 03 18:03:32.856680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:32.856623    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34"} err="failed to get container status \"61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34\": rpc error: code = NotFound desc = could not find container \"61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34\": container with ID starting with 61846055acb5e6ee965056572a5335b9fb5ca2a226783bebf5edffe9e5978b34 not found: ID does not exist"
Nov 03 18:03:33.004885 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:33.004801    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"]
Nov 03 18:03:33.054921 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:33.054884    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-s8kjj"]
Nov 03 18:03:33.125112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:33.125079    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"]
Nov 03 18:03:33.134484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:33.134448    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-r5ptp"]
Nov 03 18:03:33.992373 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:33.992342    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:34.127236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:34.127200    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" path="/var/lib/kubelet/pods/916ebdff-e9a4-47b1-ab2a-779238cd7430/volumes"
Nov 03 18:03:34.128042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:34.128015    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddace43b-4112-4594-92df-095167b2bb3f" path="/var/lib/kubelet/pods/ddace43b-4112-4594-92df-095167b2bb3f/volumes"
Nov 03 18:03:35.007103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:35.007062    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:38.774975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.774939    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"]
Nov 03 18:03:38.775720 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775705    5838 cpu_manager.go:414] "containerMap: removing container" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy"
Nov 03 18:03:38.775822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775721    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy"
Nov 03 18:03:38.775822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775752    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy"
Nov 03 18:03:38.775822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775760    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy"
Nov 03 18:03:38.775822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775787    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy"
Nov 03 18:03:38.775822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775794    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy"
Nov 03 18:03:38.775975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775912    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="916ebdff-e9a4-47b1-ab2a-779238cd7430" containerName="istio-proxy"
Nov 03 18:03:38.775975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775919    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddace43b-4112-4594-92df-095167b2bb3f" containerName="istio-proxy"
Nov 03 18:03:38.775975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.775929    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd5658a-c97e-44b2-9375-43983eb9eac1" containerName="istio-proxy"
Nov 03 18:03:38.789565 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.789530    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.827693 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.827657    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.827923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.827906    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktm4l\" (UniqueName: \"kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.827966 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.827939    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.828045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.828028    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.841046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.841007    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"]
Nov 03 18:03:38.885099 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.885055    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6ed3e95c-serving-cert\""
Nov 03 18:03:38.929702 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.929668    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.929934 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.929846    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktm4l\" (UniqueName: \"kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.929934 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.929885    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.930123 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.929950    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.930200 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:38.930140    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-serving-cert: secret "maas-api-e2e-mt-6ed3e95c-serving-cert" not found
Nov 03 18:03:38.930273 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:38.930230    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls podName:3a79e47c-937a-4adb-b3db-fcb4c900aebf nodeName:}" failed. No retries permitted until 2026-11-03 18:03:39.430187821 +0000 UTC m=+1755.440925892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls") pod "maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" (UID: "3a79e47c-937a-4adb-b3db-fcb4c900aebf") : secret "maas-api-e2e-mt-6ed3e95c-serving-cert" not found
Nov 03 18:03:38.934334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.933521    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.934334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.934294    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:38.979196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.979158    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:38.997340 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:38.997302    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktm4l\" (UniqueName: \"kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:39.148105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:39.148066    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:39.308075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:39.308038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:39.453310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:39.453219    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:39.456005 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:39.455979    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:39.705978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:39.705880    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:40.103414 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:03:40.103379    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a79e47c_937a_4adb_b3db_fcb4c900aebf.slice/crio-0617e51b1fddee62f86b73ab8cd89e28284474c5b7e6f9f551dc791cd3fe2003 WatchSource:0}: Error finding container 0617e51b1fddee62f86b73ab8cd89e28284474c5b7e6f9f551dc791cd3fe2003: Status 404 returned error can't find the container with id 0617e51b1fddee62f86b73ab8cd89e28284474c5b7e6f9f551dc791cd3fe2003
Nov 03 18:03:40.105043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:40.105009    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"]
Nov 03 18:03:40.342824 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:03:40.342780    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:03:40.672432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:40.672390    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:40.914874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:40.914841    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" event={"ID":"3a79e47c-937a-4adb-b3db-fcb4c900aebf","Type":"ContainerStarted","Data":"0617e51b1fddee62f86b73ab8cd89e28284474c5b7e6f9f551dc791cd3fe2003"}
Nov 03 18:03:41.928431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:41.928373    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" event={"ID":"3a79e47c-937a-4adb-b3db-fcb4c900aebf","Type":"ContainerStarted","Data":"b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde"}
Nov 03 18:03:41.929008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:41.928543    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:42.085443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:42.085371    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" podStartSLOduration=3.284926357 podStartE2EDuration="4.085347136s" totalImagesPullingTime="800.420779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:42.061048603 +0000 UTC m=+1758.071786712" watchObservedRunningTime="2026-11-03 18:03:42.085347136 +0000 UTC m=+1758.096085211"
Nov 03 18:03:43.118524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:43.118487    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:43.520436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:43.520392    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:43.678070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:43.678027    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:44.388276 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:44.388242    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:47.945858 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:47.945825    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:03:48.565501 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:48.565468    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:03:49.651197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:49.651151    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:51.436291 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:51.436229    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:54.808574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:54.808536    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:57.198536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.198494    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"]
Nov 03 18:03:57.199024 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.198758    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy" containerID="cri-o://53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c" gracePeriod=30
Nov 03 18:03:57.249169 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.249129    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"]
Nov 03 18:03:57.249407 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.249347    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy" containerID="cri-o://86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416" gracePeriod=30
Nov 03 18:03:57.253621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.253591    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"]
Nov 03 18:03:57.253836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.253814    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" containerID="cri-o://d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf" gracePeriod=30
Nov 03 18:03:57.301271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.301230    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"]
Nov 03 18:03:57.301586 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.301498    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy" containerID="cri-o://7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b" gracePeriod=30
Nov 03 18:03:57.720168 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.720127    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"]
Nov 03 18:03:57.724511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.724480    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"]
Nov 03 18:03:57.724785 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.724758    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy" containerID="cri-o://dda9bba30058ec149df47aed11648e3dc62b4a3391bb01b98e6cde9531358e10" gracePeriod=30
Nov 03 18:03:57.730209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.730163    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"]
Nov 03 18:03:57.730460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.730431    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy" containerID="cri-o://f54213ef7e5308e7447fefde9d0a97506fdf8d8081fb1687740dd5facc0999e5" gracePeriod=30
Nov 03 18:03:57.735823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.735792    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"]
Nov 03 18:03:57.736051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:57.736026    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy" containerID="cri-o://e82df84a1dd7bc5eeed5d5a5f4ea5fcccb3fe1480ebeb1561babd5eeeb5c6ad3" gracePeriod=30
Nov 03 18:03:58.073682 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:58.073587    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" containerID="cri-o://679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be" gracePeriod=30
Nov 03 18:03:59.862958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:03:59.862920    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:00.659969 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:00.659930    5838 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:04:00.660153 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:00.659995    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:04:02.579893 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.579861    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.610718    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.610814    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.610889    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.610978    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.611103    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.611316    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.611349    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.611382    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.611412    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs\") pod \"c86aa080-6e71-4065-a5c6-a6763a64541b\" (UID: \"c86aa080-6e71-4065-a5c6-a6763a64541b\") "
Nov 03 18:04:02.612217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.612182    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.613432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.612565    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.613432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.612579    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.613432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.612633    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.613432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.612683    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.613432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.612869    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:02.617143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.617110    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:02.617252 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.617150    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.617252 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.617182    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "kube-api-access-wnr8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.617957 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.617930    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy" pod "c86aa080-6e71-4065-a5c6-a6763a64541b" (UID: "c86aa080-6e71-4065-a5c6-a6763a64541b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.660197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.660162    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 18:04:02.713225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713190    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713309    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713380    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713518 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713420    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713518 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713460    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713518 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713467    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.713681 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713619    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713681 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713628    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.713810 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713774    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713850    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.713938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713889    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.713938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.713927    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert\") pod \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\" (UID: \"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50\") "
Nov 03 18:04:02.714503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714479    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:02.714805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714779    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnr8s\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-kube-api-access-wnr8s\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714809    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714824    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714837    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714850    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714864    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714876    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714889    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aa080-6e71-4065-a5c6-a6763a64541b-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714902    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714914    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714928    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aa080-6e71-4065-a5c6-a6763a64541b-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.714938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.714940    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.715353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.715090    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.716173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.716149    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.716596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.716570    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.716726 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.716702    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:02.716726 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.716750    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442" pod "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" (UID: "eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50"). InnerVolumeSpecName "kube-api-access-kv442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.719343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.719322    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 18:04:02.722250 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.722230    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 18:04:02.816210 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816175    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.816210 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816227    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816268    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816306    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816357    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816430    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816469    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816518    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.816576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.816578    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.817618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817319    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:02.817618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817342    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.817618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817411    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.817618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817467    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.817618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817511    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.817618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817602    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.818008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817644    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.818008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817708    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.818008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817876    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data\") pod \"12538f55-31c0-492d-8087-360d54c7ff03\" (UID: \"12538f55-31c0-492d-8087-360d54c7ff03\") "
Nov 03 18:04:02.818008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.817911    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo\") pod \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\" (UID: \"5c185a67-f484-4c14-9e8c-f560fb6c0db9\") "
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818845    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818876    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818899    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818913    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818927    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818941    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.818957    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv442\" (UniqueName: \"kubernetes.io/projected/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50-kube-api-access-kv442\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.819090    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.819161    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.819697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.819297    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:02.820327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.819776    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.820327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.819871    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.820327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.820023    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.820327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.820176    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.820327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.820214    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.821332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.821293    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:02.821870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.821835    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.822855 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.822820    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "kube-api-access-7xt9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.823148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.823112    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "kube-api-access-kj6b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.823631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.823603    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:02.823961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.823934    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy" pod "12538f55-31c0-492d-8087-360d54c7ff03" (UID: "12538f55-31c0-492d-8087-360d54c7ff03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.824074 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.824007    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:02.825136 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.825115    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy" pod "5c185a67-f484-4c14-9e8c-f560fb6c0db9" (UID: "5c185a67-f484-4c14-9e8c-f560fb6c0db9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926682    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926725    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926763    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926779    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926793    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926806    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926823    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xt9n\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-kube-api-access-7xt9n\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926836    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12538f55-31c0-492d-8087-360d54c7ff03-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926848    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926860    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926873    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12538f55-31c0-492d-8087-360d54c7ff03-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926889    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12538f55-31c0-492d-8087-360d54c7ff03-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926902    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj6b6\" (UniqueName: \"kubernetes.io/projected/5c185a67-f484-4c14-9e8c-f560fb6c0db9-kube-api-access-kj6b6\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926915    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926929    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c185a67-f484-4c14-9e8c-f560fb6c0db9-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926942    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:02.932789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:02.926958    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12538f55-31c0-492d-8087-360d54c7ff03-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.131294 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.131261    5838 generic.go:356] "Generic (PLEG): container finished" podID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerID="dda9bba30058ec149df47aed11648e3dc62b4a3391bb01b98e6cde9531358e10" exitCode=0
Nov 03 18:04:03.131488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.131341    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" event={"ID":"e52dc403-1aad-4f2a-b28c-8b14c1981c79","Type":"ContainerDied","Data":"dda9bba30058ec149df47aed11648e3dc62b4a3391bb01b98e6cde9531358e10"}
Nov 03 18:04:03.131488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.131409    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn" event={"ID":"e52dc403-1aad-4f2a-b28c-8b14c1981c79","Type":"ContainerDied","Data":"94a07a53a93dc5e186325cbec9c3cc97f848bfeb84de865ce36410b2eb7e57bb"}
Nov 03 18:04:03.131488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.131421    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 18:04:03.131488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.131435    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a07a53a93dc5e186325cbec9c3cc97f848bfeb84de865ce36410b2eb7e57bb"
Nov 03 18:04:03.131488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.131452    5838 scope.go:122] "RemoveContainer" containerID="23b9d0dfdb003ec1ed6bbe642be07455368ba018bd74e4815020aa4b30aec649"
Nov 03 18:04:03.133554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.133524    5838 generic.go:356] "Generic (PLEG): container finished" podID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerID="e82df84a1dd7bc5eeed5d5a5f4ea5fcccb3fe1480ebeb1561babd5eeeb5c6ad3" exitCode=0
Nov 03 18:04:03.133700 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.133598    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" event={"ID":"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8","Type":"ContainerDied","Data":"e82df84a1dd7bc5eeed5d5a5f4ea5fcccb3fe1480ebeb1561babd5eeeb5c6ad3"}
Nov 03 18:04:03.135699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.135678    5838 generic.go:356] "Generic (PLEG): container finished" podID="12538f55-31c0-492d-8087-360d54c7ff03" containerID="7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b" exitCode=0
Nov 03 18:04:03.135875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.135743    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"
Nov 03 18:04:03.135875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.135770    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" event={"ID":"12538f55-31c0-492d-8087-360d54c7ff03","Type":"ContainerDied","Data":"7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b"}
Nov 03 18:04:03.135875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.135825    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q" event={"ID":"12538f55-31c0-492d-8087-360d54c7ff03","Type":"ContainerDied","Data":"bd870d1767ab17194d6402abb4bd708a99d1f32903194ab9867ac6060bac2440"}
Nov 03 18:04:03.138040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.138015    5838 generic.go:356] "Generic (PLEG): container finished" podID="09369850-5e93-42a8-973d-383ba3e53363" containerID="f54213ef7e5308e7447fefde9d0a97506fdf8d8081fb1687740dd5facc0999e5" exitCode=0
Nov 03 18:04:03.138133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.138109    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" event={"ID":"09369850-5e93-42a8-973d-383ba3e53363","Type":"ContainerDied","Data":"f54213ef7e5308e7447fefde9d0a97506fdf8d8081fb1687740dd5facc0999e5"}
Nov 03 18:04:03.140081 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.140059    5838 generic.go:356] "Generic (PLEG): container finished" podID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerID="d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf" exitCode=0
Nov 03 18:04:03.140234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.140115    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"
Nov 03 18:04:03.140234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.140142    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" event={"ID":"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50","Type":"ContainerDied","Data":"d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf"}
Nov 03 18:04:03.140234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.140185    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd" event={"ID":"eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50","Type":"ContainerDied","Data":"39256c7edcdf5554e024f9ee2e4fbbf5534ca82829d9dc64ca7ce763dc94ec84"}
Nov 03 18:04:03.144698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.144670    5838 generic.go:356] "Generic (PLEG): container finished" podID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerID="53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c" exitCode=0
Nov 03 18:04:03.144885 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.144859    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"
Nov 03 18:04:03.145334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.144967    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" event={"ID":"c86aa080-6e71-4065-a5c6-a6763a64541b","Type":"ContainerDied","Data":"53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c"}
Nov 03 18:04:03.145334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.145014    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7" event={"ID":"c86aa080-6e71-4065-a5c6-a6763a64541b","Type":"ContainerDied","Data":"6438394e1cecc5c051d1cbecee807516fcc8fa8cf8dbe9260477cafe892ca639"}
Nov 03 18:04:03.164097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.162496    5838 generic.go:356] "Generic (PLEG): container finished" podID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerID="86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416" exitCode=0
Nov 03 18:04:03.164097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.162554    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" event={"ID":"5c185a67-f484-4c14-9e8c-f560fb6c0db9","Type":"ContainerDied","Data":"86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416"}
Nov 03 18:04:03.164097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.162604    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq" event={"ID":"5c185a67-f484-4c14-9e8c-f560fb6c0db9","Type":"ContainerDied","Data":"4def8c664c834720d188c8d95fc91d2af0457e86a73b712148e303872bc612eb"}
Nov 03 18:04:03.164097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.162683    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"
Nov 03 18:04:03.194680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.194656    5838 scope.go:122] "RemoveContainer" containerID="7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b"
Nov 03 18:04:03.210980 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.210957    5838 scope.go:122] "RemoveContainer" containerID="c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"
Nov 03 18:04:03.211155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.211136    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 18:04:03.215987 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.215964    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 18:04:03.223853 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.223825    5838 scope.go:122] "RemoveContainer" containerID="7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b"
Nov 03 18:04:03.224330 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:03.224306    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b\": container with ID starting with 7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b not found: ID does not exist" containerID="7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b"
Nov 03 18:04:03.224433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.224344    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b"} err="failed to get container status \"7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b\": rpc error: code = NotFound desc = could not find container \"7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b\": container with ID starting with 7d367b5e1a5f1d557caa477d9f54dcf93fd5bdfcf506f17c80379e23c386994b not found: ID does not exist"
Nov 03 18:04:03.224433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.224364    5838 scope.go:122] "RemoveContainer" containerID="c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"
Nov 03 18:04:03.224710 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:03.224687    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e\": container with ID starting with c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e not found: ID does not exist" containerID="c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"
Nov 03 18:04:03.224805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.224722    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e"} err="failed to get container status \"c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e\": rpc error: code = NotFound desc = could not find container \"c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e\": container with ID starting with c7c5de904372a4fb10440bb2b09c08c6ffe93fb3ac87e0e0141b2271e933243e not found: ID does not exist"
Nov 03 18:04:03.224805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.224784    5838 scope.go:122] "RemoveContainer" containerID="d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf"
Nov 03 18:04:03.235650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.235607    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.235828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.235693    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.235918 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.235901    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.235965 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.235928    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.236004 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.235975    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.236043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.236006    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.236089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.236054    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.236130 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.236085    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.236163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.236129    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.236214 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.236176    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token\") pod \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\" (UID: \"e52dc403-1aad-4f2a-b28c-8b14c1981c79\") "
Nov 03 18:04:03.236814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.236785    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.237251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.237229    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.237357 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.237283    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.237357 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.237309    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.237922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.237494    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.237922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.237552    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:03.240057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.238933    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:03.240057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.239417    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.240057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.239814    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.240057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.240011    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv" pod "e52dc403-1aad-4f2a-b28c-8b14c1981c79" (UID: "e52dc403-1aad-4f2a-b28c-8b14c1981c79"). InnerVolumeSpecName "kube-api-access-7k6qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.317430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.317358    5838 scope.go:122] "RemoveContainer" containerID="d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf"
Nov 03 18:04:03.319022 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:03.318274    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf\": container with ID starting with d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf not found: ID does not exist" containerID="d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf"
Nov 03 18:04:03.319022 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.318323    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf"} err="failed to get container status \"d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf\": rpc error: code = NotFound desc = could not find container \"d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf\": container with ID starting with d1321b45280744f9746c2bd3f9445c213764b57d2f2a01c2636d2e1449d6dbbf not found: ID does not exist"
Nov 03 18:04:03.319022 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.318350    5838 scope.go:122] "RemoveContainer" containerID="53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c"
Nov 03 18:04:03.337153 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337126    5838 scope.go:122] "RemoveContainer" containerID="53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c"
Nov 03 18:04:03.337522 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:03.337490    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c\": container with ID starting with 53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c not found: ID does not exist" containerID="53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c"
Nov 03 18:04:03.337664 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337529    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c"} err="failed to get container status \"53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c\": rpc error: code = NotFound desc = could not find container \"53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c\": container with ID starting with 53dba94996a58c359ad937dda579871db252279b7f225a9ecc3b6aca8caea38c not found: ID does not exist"
Nov 03 18:04:03.337664 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337557    5838 scope.go:122] "RemoveContainer" containerID="86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416"
Nov 03 18:04:03.337797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337766    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.337860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337828    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.337956 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337940    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338031 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.337978    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.338031 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338023    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338093    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.338198 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338146    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.338198 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338156    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:03.338198 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338178    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.338346 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338332    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.338400 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338360    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338405    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338439    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338550 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338454    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.338550 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338540    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338658 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338594    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.338658 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338643    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338833    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.338928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338892    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.338928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338898    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket\") pod \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\" (UID: \"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8\") "
Nov 03 18:04:03.339056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.338955    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data\") pod \"09369850-5e93-42a8-973d-383ba3e53363\" (UID: \"09369850-5e93-42a8-973d-383ba3e53363\") "
Nov 03 18:04:03.339197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.339097    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.339388 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.339370    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:03.340077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340048    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340083    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340100    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340113    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340128    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340142    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6qv\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-kube-api-access-7k6qv\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340156    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340171    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340182    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340195    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52dc403-1aad-4f2a-b28c-8b14c1981c79-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340211    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52dc403-1aad-4f2a-b28c-8b14c1981c79-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340226    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09369850-5e93-42a8-973d-383ba3e53363-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.340836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340542    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.340836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340591    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.340836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340681    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.341894 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340875    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.341894 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.340909    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.345235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.345202    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "kube-api-access-8zs9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.347219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.347183    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:03.347442 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.347418    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "kube-api-access-4ntx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.347522 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.347507    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.347624 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.347590    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.347798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.347767    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:03.349456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.349422    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy" pod "09369850-5e93-42a8-973d-383ba3e53363" (UID: "09369850-5e93-42a8-973d-383ba3e53363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.349711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.349689    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token" pod "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" (UID: "e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.413228 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.413203    5838 scope.go:122] "RemoveContainer" containerID="86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416"
Nov 03 18:04:03.413556 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:03.413525    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416\": container with ID starting with 86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416 not found: ID does not exist" containerID="86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416"
Nov 03 18:04:03.413667 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.413569    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416"} err="failed to get container status \"86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416\": rpc error: code = NotFound desc = could not find container \"86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416\": container with ID starting with 86796a8540e25d0377aa700f1995fd10f8c01e861c37b8b051b6635215102416 not found: ID does not exist"
Nov 03 18:04:03.414984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.414958    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 18:04:03.442134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442103    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442131    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09369850-5e93-42a8-973d-383ba3e53363-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442143    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zs9t\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-kube-api-access-8zs9t\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442154    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442165    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442174    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09369850-5e93-42a8-973d-383ba3e53363-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442182    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442189    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09369850-5e93-42a8-973d-383ba3e53363-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442197    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442206    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntx4\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-kube-api-access-4ntx4\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442214    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442221    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.442331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.442229    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.543130 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543096    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543145    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543174    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543220    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543290    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543372    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543414    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.543649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543590    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.543649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543621    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543640    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.543865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543668    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.543865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543711    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:03.543865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543753    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy\") pod \"6c4fc713-c468-424a-85bf-130d5552ec71\" (UID: \"6c4fc713-c468-424a-85bf-130d5552ec71\") "
Nov 03 18:04:03.544027 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.543915    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.544686 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.544659    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.544835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.544688    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.544835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.544708    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.544835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.544724    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4fc713-c468-424a-85bf-130d5552ec71-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.544835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.544758    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.545636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.545613    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:03.546064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.546037    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:03.546139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.546081    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.546217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.546199    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r" pod "6c4fc713-c468-424a-85bf-130d5552ec71" (UID: "6c4fc713-c468-424a-85bf-130d5552ec71"). InnerVolumeSpecName "kube-api-access-gdk6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:03.646285 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.646237    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdk6r\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-kube-api-access-gdk6r\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.646285 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.646278    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4fc713-c468-424a-85bf-130d5552ec71-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.646285 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.646291    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4fc713-c468-424a-85bf-130d5552ec71-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.646821 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.646306    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4fc713-c468-424a-85bf-130d5552ec71-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:03.834364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.834318    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"]
Nov 03 18:04:03.838871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.838840    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-h6tq7"]
Nov 03 18:04:03.945776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.945652    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"]
Nov 03 18:04:03.956338 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:03.956304    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-m264q"]
Nov 03 18:04:04.066901 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.066867    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"]
Nov 03 18:04:04.074082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.074042    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-j9rjq"]
Nov 03 18:04:04.120020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.119989    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12538f55-31c0-492d-8087-360d54c7ff03" path="/var/lib/kubelet/pods/12538f55-31c0-492d-8087-360d54c7ff03/volumes"
Nov 03 18:04:04.120524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.120509    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" path="/var/lib/kubelet/pods/5c185a67-f484-4c14-9e8c-f560fb6c0db9/volumes"
Nov 03 18:04:04.121008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.120994    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" path="/var/lib/kubelet/pods/c86aa080-6e71-4065-a5c6-a6763a64541b/volumes"
Nov 03 18:04:04.173110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.173073    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"
Nov 03 18:04:04.174564 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.174541    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"
Nov 03 18:04:04.174716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.174576    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j" event={"ID":"e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8","Type":"ContainerDied","Data":"76f046d4d5454494de580ace8307304398588f4dce07cecc6580761176ab6ab8"}
Nov 03 18:04:04.174716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.174647    5838 scope.go:122] "RemoveContainer" containerID="e82df84a1dd7bc5eeed5d5a5f4ea5fcccb3fe1480ebeb1561babd5eeeb5c6ad3"
Nov 03 18:04:04.177474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.177448    5838 generic.go:356] "Generic (PLEG): container finished" podID="6c4fc713-c468-424a-85bf-130d5552ec71" containerID="679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be" exitCode=0
Nov 03 18:04:04.177599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.177490    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"
Nov 03 18:04:04.177599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.177545    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" event={"ID":"6c4fc713-c468-424a-85bf-130d5552ec71","Type":"ContainerDied","Data":"679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be"}
Nov 03 18:04:04.177745 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.177599    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" event={"ID":"6c4fc713-c468-424a-85bf-130d5552ec71","Type":"ContainerDied","Data":"3589b7461cf52c962204cb372ed9a10a37560ad83d4be7eebf1b4612a4f81339"}
Nov 03 18:04:04.179272 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.179251    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"
Nov 03 18:04:04.179384 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.179265    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd" event={"ID":"09369850-5e93-42a8-973d-383ba3e53363","Type":"ContainerDied","Data":"59743312bca7eb237907161d3346887cdc2d8b83a057f6dd59771fbeb55a7d78"}
Nov 03 18:04:04.187118 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.187100    5838 scope.go:122] "RemoveContainer" containerID="679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be"
Nov 03 18:04:04.202125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.202097    5838 scope.go:122] "RemoveContainer" containerID="679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be"
Nov 03 18:04:04.203327 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:04.202549    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be\": container with ID starting with 679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be not found: ID does not exist" containerID="679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be"
Nov 03 18:04:04.203327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.202588    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be"} err="failed to get container status \"679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be\": rpc error: code = NotFound desc = could not find container \"679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be\": container with ID starting with 679eb907c9f50ec8f9a47c0bdd7474ec2dd76c0e5b7111f2ae700d0ff5ef81be not found: ID does not exist"
Nov 03 18:04:04.203327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.202614    5838 scope.go:122] "RemoveContainer" containerID="f54213ef7e5308e7447fefde9d0a97506fdf8d8081fb1687740dd5facc0999e5"
Nov 03 18:04:04.260994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.260964    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"]
Nov 03 18:04:04.261177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.261157    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" podUID="65e09abe-6887-40b5-a7c6-59c736652c25" containerName="maas-api" containerID="cri-o://510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914" gracePeriod=30
Nov 03 18:04:04.315378 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.315334    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"]
Nov 03 18:04:04.325143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.325101    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-z4bkd"]
Nov 03 18:04:04.356376 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.356342    5838 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:04:04.356556 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.356407    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:04:04.361205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.361179    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:04.407884 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.407845    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" podUID="65e09abe-6887-40b5-a7c6-59c736652c25" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.102:8443/health\": dial tcp 10.134.0.102:8443: connect: connection refused"
Nov 03 18:04:04.441938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.441884    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"]
Nov 03 18:04:04.465387 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.465296    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-8g4dp"]
Nov 03 18:04:04.603056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.603031    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:04:04.634617 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.634584    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"]
Nov 03 18:04:04.657157 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.657121    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-7g5pn"]
Nov 03 18:04:04.666201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.666169    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk\" (UniqueName: \"kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk\") pod \"65e09abe-6887-40b5-a7c6-59c736652c25\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") "
Nov 03 18:04:04.666369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.666241    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs\") pod \"65e09abe-6887-40b5-a7c6-59c736652c25\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") "
Nov 03 18:04:04.666369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.666323    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls\") pod \"65e09abe-6887-40b5-a7c6-59c736652c25\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") "
Nov 03 18:04:04.666507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.666444    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca\") pod \"65e09abe-6887-40b5-a7c6-59c736652c25\" (UID: \"65e09abe-6887-40b5-a7c6-59c736652c25\") "
Nov 03 18:04:04.666818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.666795    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca" pod "65e09abe-6887-40b5-a7c6-59c736652c25" (UID: "65e09abe-6887-40b5-a7c6-59c736652c25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:04.667520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.667500    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e09abe-6887-40b5-a7c6-59c736652c25-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.668872 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.668746    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk" pod "65e09abe-6887-40b5-a7c6-59c736652c25" (UID: "65e09abe-6887-40b5-a7c6-59c736652c25"). InnerVolumeSpecName "kube-api-access-k6zvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:04.669559 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.669518    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs" pod "65e09abe-6887-40b5-a7c6-59c736652c25" (UID: "65e09abe-6887-40b5-a7c6-59c736652c25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:04:04.670044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.669983    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls" pod "65e09abe-6887-40b5-a7c6-59c736652c25" (UID: "65e09abe-6887-40b5-a7c6-59c736652c25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:04:04.770529 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.770447    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zvk\" (UniqueName: \"kubernetes.io/projected/65e09abe-6887-40b5-a7c6-59c736652c25-kube-api-access-k6zvk\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.770529 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.770479    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.770529 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.770489    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e09abe-6887-40b5-a7c6-59c736652c25-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.879516 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.879480    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"]
Nov 03 18:04:04.885804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.885760    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-tsq6j"]
Nov 03 18:04:04.999328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:04.999291    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"]
Nov 03 18:04:05.005528 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.005497    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-259bd"]
Nov 03 18:04:05.199252 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.199216    5838 generic.go:356] "Generic (PLEG): container finished" podID="65e09abe-6887-40b5-a7c6-59c736652c25" containerID="510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914" exitCode=0
Nov 03 18:04:05.199435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.199273    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" event={"ID":"65e09abe-6887-40b5-a7c6-59c736652c25","Type":"ContainerDied","Data":"510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914"}
Nov 03 18:04:05.199435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.199297    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"
Nov 03 18:04:05.199435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.199331    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6" event={"ID":"65e09abe-6887-40b5-a7c6-59c736652c25","Type":"ContainerDied","Data":"653c02526a7593fd5fd8131e874b43f61fed3df18f5c95b5dde6217c2e2a1f93"}
Nov 03 18:04:05.199435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.199353    5838 scope.go:122] "RemoveContainer" containerID="510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914"
Nov 03 18:04:05.210851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.210831    5838 scope.go:122] "RemoveContainer" containerID="510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914"
Nov 03 18:04:05.211140 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:05.211119    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914\": container with ID starting with 510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914 not found: ID does not exist" containerID="510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914"
Nov 03 18:04:05.211246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.211148    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914"} err="failed to get container status \"510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914\": rpc error: code = NotFound desc = could not find container \"510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914\": container with ID starting with 510cfad09fe4e4e0064177b3f70e8196fecbce0a39ba656207ad2b8a0906a914 not found: ID does not exist"
Nov 03 18:04:05.388369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.388329    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"]
Nov 03 18:04:05.394859 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.394814    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e138c04-866c6db464-twlm6"]
Nov 03 18:04:05.437091 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.437058    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:05.508282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.508192    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"]
Nov 03 18:04:05.508468 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.508394    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" podUID="a96d240a-7e75-4e08-aa91-5697565a38cb" containerName="maas-api" containerID="cri-o://9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c" gracePeriod=30
Nov 03 18:04:05.658117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.658083    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:05.874159 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.874132    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:04:05.992539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.992495    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs\") pod \"a96d240a-7e75-4e08-aa91-5697565a38cb\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") "
Nov 03 18:04:05.992761 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.992698    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\") pod \"a96d240a-7e75-4e08-aa91-5697565a38cb\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") "
Nov 03 18:04:05.992841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.992793    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7\" (UniqueName: \"kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7\") pod \"a96d240a-7e75-4e08-aa91-5697565a38cb\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") "
Nov 03 18:04:05.992933 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.992919    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca\") pod \"a96d240a-7e75-4e08-aa91-5697565a38cb\" (UID: \"a96d240a-7e75-4e08-aa91-5697565a38cb\") "
Nov 03 18:04:05.993787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.993715    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca" pod "a96d240a-7e75-4e08-aa91-5697565a38cb" (UID: "a96d240a-7e75-4e08-aa91-5697565a38cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:05.996455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.996415    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7" pod "a96d240a-7e75-4e08-aa91-5697565a38cb" (UID: "a96d240a-7e75-4e08-aa91-5697565a38cb"). InnerVolumeSpecName "kube-api-access-mqdl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:05.996953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.996537    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs" pod "a96d240a-7e75-4e08-aa91-5697565a38cb" (UID: "a96d240a-7e75-4e08-aa91-5697565a38cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:04:05.997117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:05.997090    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls" pod "a96d240a-7e75-4e08-aa91-5697565a38cb" (UID: "a96d240a-7e75-4e08-aa91-5697565a38cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:04:06.099281 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.099112    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:06.099551 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.099523    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdl7\" (UniqueName: \"kubernetes.io/projected/a96d240a-7e75-4e08-aa91-5697565a38cb-kube-api-access-mqdl7\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:06.099551 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.099554    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96d240a-7e75-4e08-aa91-5697565a38cb-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:06.099698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.099569    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a96d240a-7e75-4e08-aa91-5697565a38cb-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:06.124616 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.124579    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09369850-5e93-42a8-973d-383ba3e53363" path="/var/lib/kubelet/pods/09369850-5e93-42a8-973d-383ba3e53363/volumes"
Nov 03 18:04:06.125334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.125305    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e09abe-6887-40b5-a7c6-59c736652c25" path="/var/lib/kubelet/pods/65e09abe-6887-40b5-a7c6-59c736652c25/volumes"
Nov 03 18:04:06.126126 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.126102    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" path="/var/lib/kubelet/pods/6c4fc713-c468-424a-85bf-130d5552ec71/volumes"
Nov 03 18:04:06.127385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.127366    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" path="/var/lib/kubelet/pods/e52dc403-1aad-4f2a-b28c-8b14c1981c79/volumes"
Nov 03 18:04:06.128084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.128059    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" path="/var/lib/kubelet/pods/e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8/volumes"
Nov 03 18:04:06.128634 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.128617    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" path="/var/lib/kubelet/pods/eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50/volumes"
Nov 03 18:04:06.212479 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.212440    5838 generic.go:356] "Generic (PLEG): container finished" podID="a96d240a-7e75-4e08-aa91-5697565a38cb" containerID="9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c" exitCode=0
Nov 03 18:04:06.212689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.212501    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"
Nov 03 18:04:06.212689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.212520    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" event={"ID":"a96d240a-7e75-4e08-aa91-5697565a38cb","Type":"ContainerDied","Data":"9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c"}
Nov 03 18:04:06.212689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.212571    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz" event={"ID":"a96d240a-7e75-4e08-aa91-5697565a38cb","Type":"ContainerDied","Data":"81736369ef9196d72b6ec8e1c8dbb61662fb198b24b49dea0b305fcc04f307c6"}
Nov 03 18:04:06.212689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.212601    5838 scope.go:122] "RemoveContainer" containerID="9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c"
Nov 03 18:04:06.255075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.255021    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:06.274677 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.274651    5838 scope.go:122] "RemoveContainer" containerID="9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c"
Nov 03 18:04:06.275904 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:06.275864    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c\": container with ID starting with 9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c not found: ID does not exist" containerID="9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c"
Nov 03 18:04:06.276032 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.275914    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c"} err="failed to get container status \"9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c\": rpc error: code = NotFound desc = could not find container \"9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c\": container with ID starting with 9c038eaa391c3bd96da5da4c23d518abb3d7ccf6ed400302909634c6d171197c not found: ID does not exist"
Nov 03 18:04:06.386085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.385998    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"]
Nov 03 18:04:06.389670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.389644    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-394741e0-7bd659ddc4-n2bnz"]
Nov 03 18:04:06.538093 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:06.538058    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:07.624554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:07.624503    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:07.922490 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:07.922455    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:08.088546 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.088497    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:08.120294 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.120254    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96d240a-7e75-4e08-aa91-5697565a38cb" path="/var/lib/kubelet/pods/a96d240a-7e75-4e08-aa91-5697565a38cb/volumes"
Nov 03 18:04:08.594587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.594534    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:08.834769 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.828941    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"]
Nov 03 18:04:08.834769 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.829216    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy" containerID="cri-o://cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557" gracePeriod=30
Nov 03 18:04:08.853856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.849529    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:08.853856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.853682    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"]
Nov 03 18:04:08.854086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:08.853946    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy" containerID="cri-o://bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d" gracePeriod=30
Nov 03 18:04:09.475582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:09.475518    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:09.670196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:09.670165    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:10.122493 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.122448    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"]
Nov 03 18:04:10.123511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123489    5838 cpu_manager.go:414] "containerMap: removing container" podUID="65e09abe-6887-40b5-a7c6-59c736652c25" containerName="maas-api"
Nov 03 18:04:10.123511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123511    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e09abe-6887-40b5-a7c6-59c736652c25" containerName="maas-api"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123534    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123541    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123563    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a96d240a-7e75-4e08-aa91-5697565a38cb" containerName="maas-api"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123568    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96d240a-7e75-4e08-aa91-5697565a38cb" containerName="maas-api"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123595    5838 cpu_manager.go:414] "containerMap: removing container" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123600    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123608    5838 cpu_manager.go:414] "containerMap: removing container" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123612    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123631    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123637    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123661    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123667    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123684    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123691    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123703    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy"
Nov 03 18:04:10.123705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123710    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123761    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123768    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123784    5838 cpu_manager.go:414] "containerMap: removing container" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123791    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123929    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123940    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ffcb69-1f1c-4c43-b6f4-e9ec036feaf8" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123947    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="09369850-5e93-42a8-973d-383ba3e53363" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123953    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e09abe-6887-40b5-a7c6-59c736652c25" containerName="maas-api"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123961    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4fc713-c468-424a-85bf-130d5552ec71" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123972    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96d240a-7e75-4e08-aa91-5697565a38cb" containerName="maas-api"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123984    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeba4954-d3f8-4cd5-8ffe-a58bbf3eda50" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.123995    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86aa080-6e71-4065-a5c6-a6763a64541b" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.124005    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.124017    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52dc403-1aad-4f2a-b28c-8b14c1981c79" containerName="istio-proxy"
Nov 03 18:04:10.124470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.124025    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c185a67-f484-4c14-9e8c-f560fb6c0db9" containerName="istio-proxy"
Nov 03 18:04:10.130801 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.130775    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.180364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.180328    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"]
Nov 03 18:04:10.182161 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182120    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182161 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182163    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182184    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lhd\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182238    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182270    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182291    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182363    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182648 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182492    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.182648 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.182540    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.209464 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.209424    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:10.225160 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.225131    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d9a300-openshift-default-dockercfg-tqx7m\""
Nov 03 18:04:10.284236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284191    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284236    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284264    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lhd\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284320    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284381    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284427    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284493    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284664    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284691    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.284823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.284778    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.285125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.285090    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.285238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.285096    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.285238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.285138    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.285329 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.285240    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.287756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.287716    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.287930 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.287910    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.348090 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.348056    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lhd\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.349537 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:10.349502    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:04:10.349955 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.349929    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token\") pod \"e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") " pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.399437 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.399392    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:10.435419 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.435382    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"]
Nov 03 18:04:10.436313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.436288    5838 cpu_manager.go:414] "containerMap: removing container" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy"
Nov 03 18:04:10.436313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.436311    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy"
Nov 03 18:04:10.436577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.436561    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="12538f55-31c0-492d-8087-360d54c7ff03" containerName="istio-proxy"
Nov 03 18:04:10.441824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.441799    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.452478 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.452269    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:10.518793 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.508516    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ad914d-openshift-default-dockercfg-9xgcc\""
Nov 03 18:04:10.570783 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.567921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:10.570783 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.568290    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"]
Nov 03 18:04:10.620691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.618905    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.620691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.619005    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.620691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.619062    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c79t\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.620691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.619132    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.620691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.619157    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.666430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.626959    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.666430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.627048    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.666430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.627110    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.666430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.627135    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729004    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729082    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729124    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c79t\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729188    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729215    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729316    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729367    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729420    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729445    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729903    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.729947    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.730972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.730402    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.731717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.731385    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.731717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.731550    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.745983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.737718    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.745983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.745669    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.803148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.803115    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.803640 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.803607    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c79t\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:10.972116 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:10.972022    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:11.059459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.059422    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:11.144773 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.144720    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"]
Nov 03 18:04:11.159667 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:11.159606    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12acfe5b_5879_47ae_a456_0f50375d1b53.slice/crio-0b0d0c3cbda0e51caa8e02fe9281705347b2dbf5d90100ea4fdc29e9c27fc42a WatchSource:0}: Error finding container 0b0d0c3cbda0e51caa8e02fe9281705347b2dbf5d90100ea4fdc29e9c27fc42a: Status 404 returned error can't find the container with id 0b0d0c3cbda0e51caa8e02fe9281705347b2dbf5d90100ea4fdc29e9c27fc42a
Nov 03 18:04:11.165366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.164027    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:04:11.165366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.164128    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:04:11.165366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.164178    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:04:11.264952 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.264923    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" event={"ID":"12acfe5b-5879-47ae-a456-0f50375d1b53","Type":"ContainerStarted","Data":"0b0d0c3cbda0e51caa8e02fe9281705347b2dbf5d90100ea4fdc29e9c27fc42a"}
Nov 03 18:04:11.466775 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.464698    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"]
Nov 03 18:04:11.466775 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:11.465650    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e177a1d_0f67_4cab_a5f9_19a269641e10.slice/crio-ed6ae1fdcf0c0392b1344f2e62ca895a84adadc401e7e67715df149c844814a7 WatchSource:0}: Error finding container ed6ae1fdcf0c0392b1344f2e62ca895a84adadc401e7e67715df149c844814a7: Status 404 returned error can't find the container with id ed6ae1fdcf0c0392b1344f2e62ca895a84adadc401e7e67715df149c844814a7
Nov 03 18:04:11.468191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.468144    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:04:11.468302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.468243    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:04:11.468302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:11.468290    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:04:12.274932 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.274893    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" event={"ID":"0e177a1d-0f67-4cab-a5f9-19a269641e10","Type":"ContainerStarted","Data":"c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"}
Nov 03 18:04:12.275469 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.274964    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" event={"ID":"0e177a1d-0f67-4cab-a5f9-19a269641e10","Type":"ContainerStarted","Data":"ed6ae1fdcf0c0392b1344f2e62ca895a84adadc401e7e67715df149c844814a7"}
Nov 03 18:04:12.276645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.276616    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" event={"ID":"12acfe5b-5879-47ae-a456-0f50375d1b53","Type":"ContainerStarted","Data":"83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"}
Nov 03 18:04:12.403957 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.403875    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podStartSLOduration=2.403813288 podStartE2EDuration="2.403813288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:12.401245877 +0000 UTC m=+1788.411983950" watchObservedRunningTime="2026-11-03 18:04:12.403813288 +0000 UTC m=+1788.414551361"
Nov 03 18:04:12.453324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.453294    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:12.459585 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.459553    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:12.666977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:12.666926    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podStartSLOduration=2.666908513 podStartE2EDuration="2.666908513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:12.536184765 +0000 UTC m=+1788.546922883" watchObservedRunningTime="2026-11-03 18:04:12.666908513 +0000 UTC m=+1788.677646584"
Nov 03 18:04:13.060645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.060553    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:13.066652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.066620    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:13.285500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.285468    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:13.285967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.285612    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:13.286511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.286489    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:04:13.286765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.286721    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:04:13.906043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:13.906007    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:14.227422 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.227384    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:04:14.265694 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.265663    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:04:14.295185 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.295152    5838 generic.go:356] "Generic (PLEG): container finished" podID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerID="bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d" exitCode=0
Nov 03 18:04:14.295637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.295206    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"
Nov 03 18:04:14.295637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.295241    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" event={"ID":"5bb17c87-66d7-428e-a882-aa14eaaa984a","Type":"ContainerDied","Data":"bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d"}
Nov 03 18:04:14.295637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.295297    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9" event={"ID":"5bb17c87-66d7-428e-a882-aa14eaaa984a","Type":"ContainerDied","Data":"826b1459454aa87cc5076456cd75eba06aba1e02242cdb50680e9672fbb5e5da"}
Nov 03 18:04:14.295637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.295323    5838 scope.go:122] "RemoveContainer" containerID="bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d"
Nov 03 18:04:14.297174 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.297109    5838 generic.go:356] "Generic (PLEG): container finished" podID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerID="cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557" exitCode=0
Nov 03 18:04:14.299142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.299112    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"
Nov 03 18:04:14.299296 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.299272    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" event={"ID":"c29a21d5-12e3-4bc2-9581-58ed132c851d","Type":"ContainerDied","Data":"cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557"}
Nov 03 18:04:14.299370 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.299328    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss" event={"ID":"c29a21d5-12e3-4bc2-9581-58ed132c851d","Type":"ContainerDied","Data":"5eec105f3009686ece73936c295820e6c1f512019e45269fedb1a52b29305910"}
Nov 03 18:04:14.310637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.310611    5838 scope.go:122] "RemoveContainer" containerID="bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d"
Nov 03 18:04:14.311033 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:14.310993    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d\": container with ID starting with bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d not found: ID does not exist" containerID="bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d"
Nov 03 18:04:14.311137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.311041    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d"} err="failed to get container status \"bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d\": rpc error: code = NotFound desc = could not find container \"bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d\": container with ID starting with bf3d1493fc4a05354dab506ecf0b4b8e904acabf2151f2c496623719e305ca2d not found: ID does not exist"
Nov 03 18:04:14.311137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.311058    5838 scope.go:122] "RemoveContainer" containerID="cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557"
Nov 03 18:04:14.322625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.322600    5838 scope.go:122] "RemoveContainer" containerID="cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557"
Nov 03 18:04:14.323138 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:14.323113    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557\": container with ID starting with cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557 not found: ID does not exist" containerID="cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557"
Nov 03 18:04:14.323273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.323148    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557"} err="failed to get container status \"cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557\": rpc error: code = NotFound desc = could not find container \"cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557\": container with ID starting with cf8a9663ec6b579f36c60b5b903fd2154b478062ab1aeeff60a3f5270c11d557 not found: ID does not exist"
Nov 03 18:04:14.404009 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.403972    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.404320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.404288    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.404521 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.404498    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.404678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.404661    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.405506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.404817    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.405709 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.405680    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.406647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.406627    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.407311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.407286    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.407559 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.407540    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.407768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.407750    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.407907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.407891    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.408033 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408018    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.408190 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408175    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.408316 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408301    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.408486 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408468    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.408678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408662    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.408945 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408925    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.409093 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.409077    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data\") pod \"c29a21d5-12e3-4bc2-9581-58ed132c851d\" (UID: \"c29a21d5-12e3-4bc2-9581-58ed132c851d\") "
Nov 03 18:04:14.409249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.409233    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert\") pod \"5bb17c87-66d7-428e-a882-aa14eaaa984a\" (UID: \"5bb17c87-66d7-428e-a882-aa14eaaa984a\") "
Nov 03 18:04:14.410484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.408748    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.410484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.410318    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.410484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.410371    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:14.410702 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.410633    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.410803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.409003    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.411170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411145    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:14.411329 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411304    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.411396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411335    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17c87-66d7-428e-a882-aa14eaaa984a-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.411396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411351    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.411396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411365    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.411396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411379    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.411396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411392    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a21d5-12e3-4bc2-9581-58ed132c851d-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.411653 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.411406    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.412070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.412036    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.412192 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.412142    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.412346 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.412324    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.413359 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.413308    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.415305 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.415243    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "kube-api-access-4mcrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.415434 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.415412    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:14.415522 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.415499    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.415684 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.415611    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "kube-api-access-r4wql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.415966 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.415928    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.416365 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.416344    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo" pod "5bb17c87-66d7-428e-a882-aa14eaaa984a" (UID: "5bb17c87-66d7-428e-a882-aa14eaaa984a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:14.417111 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.417080    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy" pod "c29a21d5-12e3-4bc2-9581-58ed132c851d" (UID: "c29a21d5-12e3-4bc2-9581-58ed132c851d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513516    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513561    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513576    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513591    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513604    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513617    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513632    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a21d5-12e3-4bc2-9581-58ed132c851d-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513647    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513662    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wql\" (UniqueName: \"kubernetes.io/projected/c29a21d5-12e3-4bc2-9581-58ed132c851d-kube-api-access-r4wql\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513677    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17c87-66d7-428e-a882-aa14eaaa984a-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.517768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.513690    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcrf\" (UniqueName: \"kubernetes.io/projected/5bb17c87-66d7-428e-a882-aa14eaaa984a-kube-api-access-4mcrf\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.731603 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.731501    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"]
Nov 03 18:04:14.738412 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.738375    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-gkxt9"]
Nov 03 18:04:14.849460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.849422    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"]
Nov 03 18:04:14.860060 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:14.860017    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-zx9ss"]
Nov 03 18:04:15.518593 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:15.518549    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:15.687451 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:15.687408    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:15.852809 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:15.852691    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:16.123226 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:16.123139    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" path="/var/lib/kubelet/pods/5bb17c87-66d7-428e-a882-aa14eaaa984a/volumes"
Nov 03 18:04:16.123931 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:16.123912    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" path="/var/lib/kubelet/pods/c29a21d5-12e3-4bc2-9581-58ed132c851d/volumes"
Nov 03 18:04:16.133369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:16.133314    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:16.325847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:16.325809    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:16.494442 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:16.494409    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:18.856570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:18.850080    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:19.029113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.029082    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:19.258593 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.258558    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"]
Nov 03 18:04:19.259249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.259235    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy"
Nov 03 18:04:19.259302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.259250    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy"
Nov 03 18:04:19.259302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.259294    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy"
Nov 03 18:04:19.259302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.259299    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy"
Nov 03 18:04:19.259405 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.259396    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29a21d5-12e3-4bc2-9581-58ed132c851d" containerName="istio-proxy"
Nov 03 18:04:19.259445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.259414    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb17c87-66d7-428e-a882-aa14eaaa984a" containerName="istio-proxy"
Nov 03 18:04:19.264828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.264806    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.325385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.325343    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"]
Nov 03 18:04:19.383759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.374584    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-ad914d-dockercfg-zbkq7\""
Nov 03 18:04:19.383759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.374657    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-ad914d\""
Nov 03 18:04:19.438199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.438166    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"]
Nov 03 18:04:19.444270 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.444231    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.444867 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.444840    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9l28\" (UniqueName: \"kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28\") pod \"payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.446226 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.445973    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.455488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.455428    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"]
Nov 03 18:04:19.547995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.547880    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsnhk\" (UniqueName: \"kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk\") pod \"payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.548492 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.548464    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9l28\" (UniqueName: \"kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28\") pod \"payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.549176 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.549149    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.549536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.549517    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.550781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.550545    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.620779 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.620680    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9l28\" (UniqueName: \"kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28\") pod \"payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:19.651843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.651805    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.652056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.651912    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsnhk\" (UniqueName: \"kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk\") pod \"payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.652532 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.652505    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.717841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.717805    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsnhk\" (UniqueName: \"kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk\") pod \"payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.829816 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.824005    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:19.880575 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:19.880027    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:20.235989 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.235956    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:20.291206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.291171    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"]
Nov 03 18:04:20.292715 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:20.292678    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7268918d_bd86_4dc9_9ce0_649523e28cf4.slice/crio-c628f47780a6cd53ab81c951722dfe5d2e3e937ae779edd893f238575d2b03bf WatchSource:0}: Error finding container c628f47780a6cd53ab81c951722dfe5d2e3e937ae779edd893f238575d2b03bf: Status 404 returned error can't find the container with id c628f47780a6cd53ab81c951722dfe5d2e3e937ae779edd893f238575d2b03bf
Nov 03 18:04:20.348620 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.348586    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"]
Nov 03 18:04:20.349807 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:20.349774    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133954e9_f4c9_4c11_9c4b_2e196572e9a8.slice/crio-9374908e8cb496c5ab94f6e75ea70ede8ed4406e40908fe6acc8059ca89caa87 WatchSource:0}: Error finding container 9374908e8cb496c5ab94f6e75ea70ede8ed4406e40908fe6acc8059ca89caa87: Status 404 returned error can't find the container with id 9374908e8cb496c5ab94f6e75ea70ede8ed4406e40908fe6acc8059ca89caa87
Nov 03 18:04:20.526290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.526179    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz" event={"ID":"7268918d-bd86-4dc9-9ce0-649523e28cf4","Type":"ContainerStarted","Data":"78ed7f4d0fe12eed90534fee31b8b0e44f719c29bf3891b393d268a9fec254f9"}
Nov 03 18:04:20.526290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.526256    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:20.526290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.526279    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz" event={"ID":"7268918d-bd86-4dc9-9ce0-649523e28cf4","Type":"ContainerStarted","Data":"c628f47780a6cd53ab81c951722dfe5d2e3e937ae779edd893f238575d2b03bf"}
Nov 03 18:04:20.528095 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.528061    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj" event={"ID":"133954e9-f4c9-4c11-9c4b-2e196572e9a8","Type":"ContainerStarted","Data":"7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db"}
Nov 03 18:04:20.528261 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.528107    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj" event={"ID":"133954e9-f4c9-4c11-9c4b-2e196572e9a8","Type":"ContainerStarted","Data":"9374908e8cb496c5ab94f6e75ea70ede8ed4406e40908fe6acc8059ca89caa87"}
Nov 03 18:04:20.528261 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.528155    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:20.650458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.650383    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz" podStartSLOduration=1.65036299 podStartE2EDuration="1.65036299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:20.647879783 +0000 UTC m=+1796.658617893" watchObservedRunningTime="2026-11-03 18:04:20.65036299 +0000 UTC m=+1796.661101062"
Nov 03 18:04:20.767820 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:20.767708    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj" podStartSLOduration=1.767693096 podStartE2EDuration="1.767693096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:20.766952063 +0000 UTC m=+1796.777690136" watchObservedRunningTime="2026-11-03 18:04:20.767693096 +0000 UTC m=+1796.778431158"
Nov 03 18:04:23.021199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:23.021143    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:23.513758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:23.507451    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:23.892595 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:23.892560    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:04:23.892811 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:23.892618    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:04:24.277882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:24.269047    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:24.516315 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:24.516280    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:04:24.517320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:24.517293    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:04:24.675630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:24.675591    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:24.737012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:24.736974    5838 scope.go:122] "RemoveContainer" containerID="dda9bba30058ec149df47aed11648e3dc62b4a3391bb01b98e6cde9531358e10"
Nov 03 18:04:26.529703 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:26.529637    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:26.708908 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:26.708862    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:26.933783 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:26.931154    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:28.414885 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:28.414853    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"]
Nov 03 18:04:28.415328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:28.415073    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy" containerID="cri-o://185e2bde2f9b4dac19c675f2cbfbf6d19107c374a0c8e9d90b5bdb8a5ce035d1" gracePeriod=30
Nov 03 18:04:28.465636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:28.465600    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"]
Nov 03 18:04:28.465912 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:28.465881    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy" containerID="cri-o://bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac" gracePeriod=30
Nov 03 18:04:30.724972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:30.724921    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:30.901167 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:30.901129    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:31.137644 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:31.137557    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:31.635539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:31.635508    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:04:31.635795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:31.635582    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:04:32.569099 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:32.569009    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:33.744505 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.744461    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.750770    5838 generic.go:356] "Generic (PLEG): container finished" podID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerID="185e2bde2f9b4dac19c675f2cbfbf6d19107c374a0c8e9d90b5bdb8a5ce035d1" exitCode=0
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.750935    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" event={"ID":"5415ca1c-de9f-41e4-90ff-7ad756afedb2","Type":"ContainerDied","Data":"185e2bde2f9b4dac19c675f2cbfbf6d19107c374a0c8e9d90b5bdb8a5ce035d1"}
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.753168    5838 generic.go:356] "Generic (PLEG): container finished" podID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerID="bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac" exitCode=0
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.753239    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" event={"ID":"6a026b47-8024-43da-a7aa-7669e1712b0b","Type":"ContainerDied","Data":"bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac"}
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.753284    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46" event={"ID":"6a026b47-8024-43da-a7aa-7669e1712b0b","Type":"ContainerDied","Data":"c2a41796af640339e631c2c8994e9e8b37f46505ee69b0a0db47c7bc68e59cdc"}
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.753337    5838 scope.go:122] "RemoveContainer" containerID="bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac"
Nov 03 18:04:33.754541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.753522    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"
Nov 03 18:04:33.780135 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.780113    5838 scope.go:122] "RemoveContainer" containerID="bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac"
Nov 03 18:04:33.780395 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:33.780373    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac\": container with ID starting with bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac not found: ID does not exist" containerID="bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac"
Nov 03 18:04:33.780493 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.780410    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac"} err="failed to get container status \"bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac\": rpc error: code = NotFound desc = could not find container \"bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac\": container with ID starting with bdfb5256aa12e7995264b23ec16db31ec8628bbe52a097954dacd1277515c3ac not found: ID does not exist"
Nov 03 18:04:33.783534 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.783508    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 18:04:33.848934 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.848896    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849146 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849037    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849146 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849100    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849157    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849225    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849387 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849274    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849413    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849498 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849452    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849498 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849452    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.849498 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849491    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket\") pod \"6a026b47-8024-43da-a7aa-7669e1712b0b\" (UID: \"6a026b47-8024-43da-a7aa-7669e1712b0b\") "
Nov 03 18:04:33.849642 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849500    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:33.849642 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849567    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.849724 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849649    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.849874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.849853    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.850408 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.850383    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.850547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.850409    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.850547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.850426    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.850547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.850439    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a026b47-8024-43da-a7aa-7669e1712b0b-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.850547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.850453    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.851930 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.851903    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:33.851930 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.851909    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.852278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.852250    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:33.852524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.852503    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt" pod "6a026b47-8024-43da-a7aa-7669e1712b0b" (UID: "6a026b47-8024-43da-a7aa-7669e1712b0b"). InnerVolumeSpecName "kube-api-access-2fhgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:33.951705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.951603    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.951705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.951695    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.951751    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.951807    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.951927    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952180 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952012    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952180 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952044    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.952180 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952066    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952194    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.952337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952271    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token\") pod \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\" (UID: \"5415ca1c-de9f-41e4-90ff-7ad756afedb2\") "
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952492    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.952816    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953025    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953179    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953195    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953204    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953216    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953227    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953240    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953250    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhgt\" (UniqueName: \"kubernetes.io/projected/6a026b47-8024-43da-a7aa-7669e1712b0b-kube-api-access-2fhgt\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953262    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a026b47-8024-43da-a7aa-7669e1712b0b-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:33.953334 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.953311    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:33.958147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.958107    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:33.958268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.958178    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:33.958917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.958892    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "kube-api-access-6chg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:33.960806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:33.960781    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy" pod "5415ca1c-de9f-41e4-90ff-7ad756afedb2" (UID: "5415ca1c-de9f-41e4-90ff-7ad756afedb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:34.055103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.055059    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chg6\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-kube-api-access-6chg6\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:34.055103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.055092    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:34.055103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.055104    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:34.055347 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.055113    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:34.055347 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.055122    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5415ca1c-de9f-41e4-90ff-7ad756afedb2-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:34.110230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.110191    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:34.222535 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.222453    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"]
Nov 03 18:04:34.229055 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.229026    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mql46"]
Nov 03 18:04:34.271992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.271956    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:34.502878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.502793    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"]
Nov 03 18:04:34.503907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.503886    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy"
Nov 03 18:04:34.503964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.503911    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy"
Nov 03 18:04:34.504006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.503967    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy"
Nov 03 18:04:34.504006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.503976    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy"
Nov 03 18:04:34.504165 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.504153    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" containerName="istio-proxy"
Nov 03 18:04:34.504200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.504179    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" containerName="istio-proxy"
Nov 03 18:04:34.516794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.516755    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.572281 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.572248    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-62aead-dockercfg-5sxg5\""
Nov 03 18:04:34.573124 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.573097    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-62aead\""
Nov 03 18:04:34.573360 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.573334    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"]
Nov 03 18:04:34.685024 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.684983    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.685443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.685415    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnlh\" (UniqueName: \"kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh\") pod \"payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.788286 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.788185    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.789045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.789016    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnlh\" (UniqueName: \"kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh\") pod \"payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.790554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.790525    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.825416 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.825377    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"]
Nov 03 18:04:34.833468 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.832760    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"
Nov 03 18:04:34.843706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.842988    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"]
Nov 03 18:04:34.843706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.843041    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9" event={"ID":"5415ca1c-de9f-41e4-90ff-7ad756afedb2","Type":"ContainerDied","Data":"6a49f29a67f8d9720519775a457b6b442573f3d1d53836da01f7ac424b44c3d4"}
Nov 03 18:04:34.843706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.843073    5838 scope.go:122] "RemoveContainer" containerID="185e2bde2f9b4dac19c675f2cbfbf6d19107c374a0c8e9d90b5bdb8a5ce035d1"
Nov 03 18:04:34.843706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.843139    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:34.858207 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.857384    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnlh\" (UniqueName: \"kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh\") pod \"payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:34.992273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.992231    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2zs\" (UniqueName: \"kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs\") pod \"payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:34.992481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:34.992308    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.093675 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.093584    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2zs\" (UniqueName: \"kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs\") pod \"payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.093675 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.093642    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.094441 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.094413    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.123028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.122993    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"]
Nov 03 18:04:35.129698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.129659    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-z7mx9"]
Nov 03 18:04:35.136570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.136541    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:35.158510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.158479    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2zs\" (UniqueName: \"kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs\") pod \"payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.179718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.179681    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.529562 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:35.529507    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e9aa53_7fd0_4178_8c5c_e0d65fd66f99.slice/crio-9ea0d667285d74fd91212de648f64b33f76c2d258ea96db2934489724e8b2d63 WatchSource:0}: Error finding container 9ea0d667285d74fd91212de648f64b33f76c2d258ea96db2934489724e8b2d63: Status 404 returned error can't find the container with id 9ea0d667285d74fd91212de648f64b33f76c2d258ea96db2934489724e8b2d63
Nov 03 18:04:35.530888 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.530857    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"]
Nov 03 18:04:35.597316 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:35.597277    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c89157_87af_4af2_a659_c6bac4bec72b.slice/crio-44813ff090f6e2a0e38c0c99e69e20b3d49d661e4dbee5489571fefc58eea6cf WatchSource:0}: Error finding container 44813ff090f6e2a0e38c0c99e69e20b3d49d661e4dbee5489571fefc58eea6cf: Status 404 returned error can't find the container with id 44813ff090f6e2a0e38c0c99e69e20b3d49d661e4dbee5489571fefc58eea6cf
Nov 03 18:04:35.599550 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.599516    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"]
Nov 03 18:04:35.845642 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.845529    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps" event={"ID":"23c89157-87af-4af2-a659-c6bac4bec72b","Type":"ContainerStarted","Data":"fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363"}
Nov 03 18:04:35.845642 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.845637    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:35.846189 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.845666    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps" event={"ID":"23c89157-87af-4af2-a659-c6bac4bec72b","Type":"ContainerStarted","Data":"44813ff090f6e2a0e38c0c99e69e20b3d49d661e4dbee5489571fefc58eea6cf"}
Nov 03 18:04:35.848679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.848655    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc" event={"ID":"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99","Type":"ContainerStarted","Data":"eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84"}
Nov 03 18:04:35.848800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.848694    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc" event={"ID":"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99","Type":"ContainerStarted","Data":"9ea0d667285d74fd91212de648f64b33f76c2d258ea96db2934489724e8b2d63"}
Nov 03 18:04:35.848800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.848761    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:35.968082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:35.968024    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps" podStartSLOduration=1.968008004 podStartE2EDuration="1.968008004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:35.965862799 +0000 UTC m=+1811.976600888" watchObservedRunningTime="2026-11-03 18:04:35.968008004 +0000 UTC m=+1811.978746075"
Nov 03 18:04:36.092113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:36.092046    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc" podStartSLOduration=2.09202259 podStartE2EDuration="2.09202259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:36.086025593 +0000 UTC m=+1812.096763670" watchObservedRunningTime="2026-11-03 18:04:36.09202259 +0000 UTC m=+1812.102760664"
Nov 03 18:04:36.123433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:36.123340    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5415ca1c-de9f-41e4-90ff-7ad756afedb2" path="/var/lib/kubelet/pods/5415ca1c-de9f-41e4-90ff-7ad756afedb2/volumes"
Nov 03 18:04:36.124653 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:36.124620    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a026b47-8024-43da-a7aa-7669e1712b0b" path="/var/lib/kubelet/pods/6a026b47-8024-43da-a7aa-7669e1712b0b/volumes"
Nov 03 18:04:38.065801 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:38.065746    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:40.341588 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:04:40.341539    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:04:41.039515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:41.039481    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:41.264140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:41.264106    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:42.760878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:42.760835    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:04:43.370321 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:43.370288    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:46.860747 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:46.860694    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:04:46.861298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:46.860994    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:04:46.957619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:46.957584    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:47.648758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:47.640812    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:47.802531 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:47.802487    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:48.473495 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:48.473446    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:48.915944 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:48.915910    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:51.007089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.007049    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"]
Nov 03 18:04:51.011414 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.011394    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.107443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.107372    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:51.116585 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.116540    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6ed3e95c\"/\"openshift-service-ca.crt\""
Nov 03 18:04:51.117333 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.117308    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6ed3e95c\"/\"kube-root-ca.crt\""
Nov 03 18:04:51.117515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.117460    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6ed3e95c\"/\"default-dockercfg-6k9pj\""
Nov 03 18:04:51.117620 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.117528    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6ed3e95c\"/\"e2e-lifecycle-model-6ed3e95c-kserve-self-signed-certs\""
Nov 03 18:04:51.121247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.121221    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"]
Nov 03 18:04:51.169520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.169467    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs8qx\" (UniqueName: \"kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.169717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.169562    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.169717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.169603    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.169717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.169652    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.169905 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.169794    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.170035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.170009    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.271489 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.271400    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs8qx\" (UniqueName: \"kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.271489 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.271479    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.271797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.271508    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.271797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.271552    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.271797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.271653    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.271797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.271724    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.272041 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.272014    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.272101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.272044    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.272162 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.272105    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.274723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.274691    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.278929 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.278897    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.339133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.339101    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs8qx\" (UniqueName: \"kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx\") pod \"e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") " pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:51.628821 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:51.626605    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:52.038014 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:52.037984    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"]
Nov 03 18:04:52.039140 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:52.039112    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode07e2162_258d_4d72_af74_4ccd27078ef2.slice/crio-2036ba4ed5c5e3f55fd966b9ce16e44f29718b10aa8ee867eb1e3aba7681b0ba WatchSource:0}: Error finding container 2036ba4ed5c5e3f55fd966b9ce16e44f29718b10aa8ee867eb1e3aba7681b0ba: Status 404 returned error can't find the container with id 2036ba4ed5c5e3f55fd966b9ce16e44f29718b10aa8ee867eb1e3aba7681b0ba
Nov 03 18:04:52.148129 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:52.148091    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:52.322813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:52.322682    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:52.494880 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:52.494847    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:53.004276 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:53.004235    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" event={"ID":"e07e2162-258d-4d72-af74-4ccd27078ef2","Type":"ContainerStarted","Data":"f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6"}
Nov 03 18:04:53.004520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:53.004294    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" event={"ID":"e07e2162-258d-4d72-af74-4ccd27078ef2","Type":"ContainerStarted","Data":"2036ba4ed5c5e3f55fd966b9ce16e44f29718b10aa8ee867eb1e3aba7681b0ba"}
Nov 03 18:04:53.134138 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:53.134079    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" podStartSLOduration=3.134062275 podStartE2EDuration="3.134062275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:53.132187307 +0000 UTC m=+1829.142925379" watchObservedRunningTime="2026-11-03 18:04:53.134062275 +0000 UTC m=+1829.144800404"
Nov 03 18:04:53.627329 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:53.627290    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:04:54.504357 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:54.504318    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:57.303715 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.303676    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:57.543157 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.543055    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:57.752212 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.752175    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:57.874838 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.874799    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"]
Nov 03 18:04:57.881253 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.881214    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:57.949424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.949386    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:57.963806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.961034    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"]
Nov 03 18:04:57.988855 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.988628    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8dfc64-dockercfg-fjws9\""
Nov 03 18:04:57.988855 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:57.988786    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8dfc64\""
Nov 03 18:04:58.018605 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.018503    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.018825 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.018744    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mmw\" (UniqueName: \"kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw\") pod \"payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.119886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.119852    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mmw\" (UniqueName: \"kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw\") pod \"payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.120101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.119999    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.120919 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.120897    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.190263 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.186913    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mmw\" (UniqueName: \"kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw\") pod \"payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.260278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.260235    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:58.647573 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:58.647537    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"]
Nov 03 18:04:58.652322 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:04:58.652283    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dd4294_32ab_48cc_904a_00dade8a2d63.slice/crio-6c1b1f19f3c4161bb4b0426ebe0419ab201443354cff8ac1fcc855963961ead5 WatchSource:0}: Error finding container 6c1b1f19f3c4161bb4b0426ebe0419ab201443354cff8ac1fcc855963961ead5: Status 404 returned error can't find the container with id 6c1b1f19f3c4161bb4b0426ebe0419ab201443354cff8ac1fcc855963961ead5
Nov 03 18:04:59.080860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:59.080759    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" event={"ID":"97dd4294-32ab-48cc-904a-00dade8a2d63","Type":"ContainerStarted","Data":"ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca"}
Nov 03 18:04:59.080860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:59.080817    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" event={"ID":"97dd4294-32ab-48cc-904a-00dade8a2d63","Type":"ContainerStarted","Data":"6c1b1f19f3c4161bb4b0426ebe0419ab201443354cff8ac1fcc855963961ead5"}
Nov 03 18:04:59.081108 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:59.080915    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:04:59.255455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:59.255381    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" podStartSLOduration=2.255360102 podStartE2EDuration="2.255360102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:59.250220109 +0000 UTC m=+1835.260958183" watchObservedRunningTime="2026-11-03 18:04:59.255360102 +0000 UTC m=+1835.266098175"
Nov 03 18:04:59.380341 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:59.380227    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:59.564708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:04:59.563632    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:00.450886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:00.450855    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:01.628190 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:01.628159    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:05:01.645668 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:01.645637    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:05:02.119692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:02.119662    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:05:05.906032 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:05.905989    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:10.092751 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:10.092699    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:05:10.349287 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:05:10.349190    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:05:10.890503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:10.890458    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:11.059233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:11.059197    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:11.260995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:11.260897    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:13.194992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:13.194956    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"]
Nov 03 18:05:13.195503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:13.195178    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy" containerID="cri-o://c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25" gracePeriod=30
Nov 03 18:05:14.253701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:14.253668    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:15.467532 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:15.467482    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:15.806265 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:15.806172    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:16.259566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:16.259528    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:16.442074 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:16.442038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:16.800873 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:16.800841    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:17.686822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:17.686776    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:18.912310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:18.912280    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 18:05:19.058820 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.058688    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.059213 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.059177    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.059412 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.059383    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.059625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.059607    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.059881 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.059852    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.060049 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.060031    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.060302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.060276    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.060524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.060505    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.060701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.060674    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket\") pod \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\" (UID: \"5c2d2de9-bde5-4783-9ffb-afb416501fd6\") "
Nov 03 18:05:19.062113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.062088    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:05:19.062457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.062437    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:05:19.062832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.062811    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:05:19.067427 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.067405    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:05:19.072962 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.072934    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:19.085197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.085143    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "kube-api-access-dhx72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:19.085520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.085498    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:19.085770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.085719    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:05:19.085992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.085962    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo" pod "5c2d2de9-bde5-4783-9ffb-afb416501fd6" (UID: "5c2d2de9-bde5-4783-9ffb-afb416501fd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.173967    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174003    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174022    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174039    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174054    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx72\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-kube-api-access-dhx72\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174069    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174085    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174102    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.185085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.174116    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2d2de9-bde5-4783-9ffb-afb416501fd6-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.315965 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.315694    5838 generic.go:356] "Generic (PLEG): container finished" podID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerID="c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25" exitCode=0
Nov 03 18:05:19.316399 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.316369    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" event={"ID":"5c2d2de9-bde5-4783-9ffb-afb416501fd6","Type":"ContainerDied","Data":"c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25"}
Nov 03 18:05:19.316625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.316603    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns" event={"ID":"5c2d2de9-bde5-4783-9ffb-afb416501fd6","Type":"ContainerDied","Data":"fb2946c7a2cbb555a63d02445a70875eceb51a0794483614cbee8bbae3403b72"}
Nov 03 18:05:19.316804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.316786    5838 scope.go:122] "RemoveContainer" containerID="c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25"
Nov 03 18:05:19.317110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.317091    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"
Nov 03 18:05:19.352923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.352807    5838 scope.go:122] "RemoveContainer" containerID="c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25"
Nov 03 18:05:19.353363 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:05:19.353318    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25\": container with ID starting with c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25 not found: ID does not exist" containerID="c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25"
Nov 03 18:05:19.353564 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.353533    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25"} err="failed to get container status \"c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25\": rpc error: code = NotFound desc = could not find container \"c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25\": container with ID starting with c2aeb5e23358297c7aa27e2d05954154d40185fba1a5a278d6625c4e22985e25 not found: ID does not exist"
Nov 03 18:05:19.503553 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.503512    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"]
Nov 03 18:05:19.510093 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:19.510062    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-tblns"]
Nov 03 18:05:20.128824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:20.128775    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" path="/var/lib/kubelet/pods/5c2d2de9-bde5-4783-9ffb-afb416501fd6/volumes"
Nov 03 18:05:20.971679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:20.971628    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:21.157381 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:21.157343    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:21.326156 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:21.326046    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:22.527352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:22.527321    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:25.821238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:25.821144    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:25.991427 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:25.991395    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:27.609009 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:27.608963    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:28.453992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:28.453950    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:33.184900 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:33.184859    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:33.836544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:33.836504    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:38.519485 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.519450    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"]
Nov 03 18:05:38.520215 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.520198    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy"
Nov 03 18:05:38.520264 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.520216    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy"
Nov 03 18:05:38.520363 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.520354    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2d2de9-bde5-4783-9ffb-afb416501fd6" containerName="istio-proxy"
Nov 03 18:05:38.525244 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.525219    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.628506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.628459    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.628719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.628548    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.628910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.628879    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.629127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.629102    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7c2\" (UniqueName: \"kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.630296 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.630266    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-62aead-serving-cert\""
Nov 03 18:05:38.637004 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.636971    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"]
Nov 03 18:05:38.723750 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.723704    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:38.730150 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.730123    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.730306 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.730261    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.730375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.730347    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7c2\" (UniqueName: \"kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.730432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.730383    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.730521 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:05:38.730504    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-serving-cert: secret "maas-api-e2e-shared-b-w0-62aead-serving-cert" not found
Nov 03 18:05:38.730587 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:05:38.730567    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls podName:041f2c85-51cd-40e8-9931-625774cbe220 nodeName:}" failed. No retries permitted until 2026-11-03 18:05:39.230550413 +0000 UTC m=+1875.241288464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls") pod "maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" (UID: "041f2c85-51cd-40e8-9931-625774cbe220") : secret "maas-api-e2e-shared-b-w0-62aead-serving-cert" not found
Nov 03 18:05:38.730831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.730810    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.732955 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.732931    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.796686 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.796604    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7c2\" (UniqueName: \"kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:38.881613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:38.881576    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:39.105295 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:39.105202    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:39.237119 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:39.237080    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:39.240202 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:39.240173    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:39.441422 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:39.441382    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:39.833619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:39.833576    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"]
Nov 03 18:05:39.834056 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:05:39.833923    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041f2c85_51cd_40e8_9931_625774cbe220.slice/crio-2ae02a72b6e2850614b8e10becc5b18741d7abda1724bf942af7fad20d302f0d WatchSource:0}: Error finding container 2ae02a72b6e2850614b8e10becc5b18741d7abda1724bf942af7fad20d302f0d: Status 404 returned error can't find the container with id 2ae02a72b6e2850614b8e10becc5b18741d7abda1724bf942af7fad20d302f0d
Nov 03 18:05:40.341007 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:05:40.340967    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:05:40.581802 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:40.581760    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" event={"ID":"041f2c85-51cd-40e8-9931-625774cbe220","Type":"ContainerStarted","Data":"2ae02a72b6e2850614b8e10becc5b18741d7abda1724bf942af7fad20d302f0d"}
Nov 03 18:05:41.387800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:41.387756    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:41.591222 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:41.591185    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" event={"ID":"041f2c85-51cd-40e8-9931-625774cbe220","Type":"ContainerStarted","Data":"58a482dc2a7b39bc21d85ec8d83841d6078e21dc8f1275606a9dec50c75dd354"}
Nov 03 18:05:41.591417 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:41.591271    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:41.715450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:41.715385    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" podStartSLOduration=2.910767789 podStartE2EDuration="3.715368902s" totalImagesPullingTime="804.601113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:41.711881864 +0000 UTC m=+1877.722619949" watchObservedRunningTime="2026-11-03 18:05:41.715368902 +0000 UTC m=+1877.726106973"
Nov 03 18:05:44.108757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:44.107264    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:47.605968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:47.605938    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:05:49.111248 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:49.111215    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:54.114566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:54.114525    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:59.138429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:05:59.138391    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:02.536950 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:02.536873    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:06:04.223058 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:04.222974    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:04.396984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:04.396945    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:04.601506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:04.601396    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:06.786765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:06.785894    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:06.831897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:06.831839    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:06:06.931772 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:06.929859    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.132450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.132319    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.311762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.310079    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.383967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.383861    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:07.384254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.384222    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.440092 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.440024    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.574888 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.574836    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.789889 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.789831    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 18:06:07.790917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.790892    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 18:06:07.836315 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.836266    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:07.837174 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:07.837150    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 18:06:08.469857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.469791    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:08.510757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.470697    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 18:06:08.801969 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.801859    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 18:06:08.841420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.841375    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:06:08.977209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.940827    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:08.977209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.974445    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:08.977209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:08.974514    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:09.018907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:09.010441    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 18:06:09.055972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:09.055885    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 18:06:09.137424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:09.137383    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:09.160755 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:09.156978    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 18:06:09.177686 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:09.177592    5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:10.394971 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:06:10.394903    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:06:10.508892 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:10.508853    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:14.286468 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:14.286427    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:14.286908 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:14.286539    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:14.326964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:14.326920    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:14.327174 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:14.327000    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:15.669003 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.668969    5838 generic.go:356] "Generic (PLEG): container finished" podID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerID="c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a" exitCode=0
Nov 03 18:06:15.669568 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.669043    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" event={"ID":"0e177a1d-0f67-4cab-a5f9-19a269641e10","Type":"ContainerDied","Data":"c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"}
Nov 03 18:06:15.669660 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.669592    5838 scope.go:122] "RemoveContainer" containerID="c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"
Nov 03 18:06:15.670900 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.670878    5838 generic.go:356] "Generic (PLEG): container finished" podID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerID="83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008" exitCode=0
Nov 03 18:06:15.670991 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.670957    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" event={"ID":"12acfe5b-5879-47ae-a456-0f50375d1b53","Type":"ContainerDied","Data":"83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"}
Nov 03 18:06:15.671410 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.671394    5838 scope.go:122] "RemoveContainer" containerID="83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"
Nov 03 18:06:15.671643 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.671604    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:15.671719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.671701    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:15.671821 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.671778    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:15.673146 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.673105    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:15.673232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.673194    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:15.673275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.673238    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:15.679636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:15.679609    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:16.688985 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:16.688946    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" event={"ID":"0e177a1d-0f67-4cab-a5f9-19a269641e10","Type":"ContainerStarted","Data":"ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95"}
Nov 03 18:06:16.691234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:16.691208    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" event={"ID":"12acfe5b-5879-47ae-a456-0f50375d1b53","Type":"ContainerStarted","Data":"0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0"}
Nov 03 18:06:17.059772 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:17.059637    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:06:17.060507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:17.060481    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.060722 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:17.060689    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 18:06:17.455247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:17.455202    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:06:17.455639 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:17.455615    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.455884 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:17.455857    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 18:06:18.061195 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:18.061153    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.062333 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:18.061908    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 18:06:18.460759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:18.459226    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.460759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:18.459290    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 18:06:19.062294 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:19.062246    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.062716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:19.062322    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 18:06:19.457416 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:19.457373    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.457695 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:19.457443    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 18:06:20.060715 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:20.060672    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.061022 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:20.060971    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 18:06:20.458982 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:20.458945    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.459586 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:20.459549    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 18:06:21.060280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:21.060236    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.060594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:21.060563    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 18:06:21.212803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:21.212721    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:21.457341 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:21.457237    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.457341 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:21.457316    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 18:06:22.060076 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:22.060042    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:22.060711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:22.060675    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 18:06:22.514048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:22.514010    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:06:22.514538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:22.514478    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:06:22.515454 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:22.515418    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:06:23.064330 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:23.064291    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:06:23.064853 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:23.064771    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:06:23.065683 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:23.065655    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:06:26.296772 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:26.296721    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:31.531770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:31.531652    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:33.835155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:33.835116    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:34.348870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:34.348830    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:35.129027 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:35.128988    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:35.412924 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:35.412867    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:36.864353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.864313    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"]
Nov 03 18:06:36.869560 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.869531    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:36.928845 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.928799    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"]
Nov 03 18:06:36.930362 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.930289    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:36.930549 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.930429    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:36.930656 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.930630    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwx8\" (UniqueName: \"kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:36.930926 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.930894    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:36.972489 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:36.972463    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b05459-serving-cert\""
Nov 03 18:06:37.032657 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.032619    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwx8\" (UniqueName: \"kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.032883 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.032720    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.032941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.032912    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.033002 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.032971    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.033087 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:06:37.033063    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-serving-cert: secret "maas-api-e2e-api-a-b05459-serving-cert" not found
Nov 03 18:06:37.033170 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:06:37.033157    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls podName:c0022528-80fa-445f-8ff6-6b40803cb91d nodeName:}" failed. No retries permitted until 2026-11-03 18:06:37.53313798 +0000 UTC m=+1933.543876050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls") pod "maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" (UID: "c0022528-80fa-445f-8ff6-6b40803cb91d") : secret "maas-api-e2e-api-a-b05459-serving-cert" not found
Nov 03 18:06:37.033948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.033920    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.035647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.035627    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.074040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.074000    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:37.099201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.099162    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwx8\" (UniqueName: \"kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.234186 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.234151    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:37.392257 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.392225    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:37.546634 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.546533    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.549193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.549165    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\") pod \"maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:37.785498 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:37.785460    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:38.190195 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:38.190141    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"]
Nov 03 18:06:38.190569 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:06:38.190375    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0022528_80fa_445f_8ff6_6b40803cb91d.slice/crio-3f5b0d1f9cb41c123d8a15cdcd88e7f70578e982caf99c396088c442b7df88d3 WatchSource:0}: Error finding container 3f5b0d1f9cb41c123d8a15cdcd88e7f70578e982caf99c396088c442b7df88d3: Status 404 returned error can't find the container with id 3f5b0d1f9cb41c123d8a15cdcd88e7f70578e982caf99c396088c442b7df88d3
Nov 03 18:06:38.977840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:38.977790    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" event={"ID":"c0022528-80fa-445f-8ff6-6b40803cb91d","Type":"ContainerStarted","Data":"3f5b0d1f9cb41c123d8a15cdcd88e7f70578e982caf99c396088c442b7df88d3"}
Nov 03 18:06:38.995979 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:38.995942    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:39.522104 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:39.522074    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:39.735594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:39.735557    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:39.952380 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:39.952343    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:40.081889 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:40.081847    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" event={"ID":"c0022528-80fa-445f-8ff6-6b40803cb91d","Type":"ContainerStarted","Data":"cadd1d483b87974e46b46601fd87849db98653b1c1a191ac361c49c42977fdcf"}
Nov 03 18:06:40.082282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:40.082259    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:40.214910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:40.214020    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" podStartSLOduration=3.337065108 podStartE2EDuration="4.214002853s" totalImagesPullingTime="876.937745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:40.213868268 +0000 UTC m=+1936.224606341" watchObservedRunningTime="2026-11-03 18:06:40.214002853 +0000 UTC m=+1936.224740924"
Nov 03 18:06:40.335329 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:40.335292    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:40.342637 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:06:40.342599    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:06:40.567821 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:40.567392    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:41.746992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:41.746957    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:43.085543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:43.085502    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:45.053670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:45.053568    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:45.245619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:45.245583    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:45.441304 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:45.441265    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:06:46.182392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:46.182358    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:46.378050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:46.378013    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:46.547530 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:46.547446    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:47.102395 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:47.102365    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:06:49.103798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:49.103763    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:54.418071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:54.418036    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:54.890354 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:54.890325    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:55.090764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:55.090677    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:59.266219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:59.266180    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:59.726111 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:06:59.726072    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:05.027610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:05.027570    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:07.098459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:07.098417    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:09.946762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:09.946421    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:10.339886 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:07:10.339799    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:07:10.584703 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:10.584663    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:15.298071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:15.298026    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:15.745282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:15.745238    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:16.366825 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:16.366772    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:16.579353 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:16.579315    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:20.566552 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:20.566514    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:20.732849 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:20.732810    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:20.897348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:20.897308    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:21.072194 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:21.072155    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:23.877345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:23.877314    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:24.065428 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.065329    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:24.439028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.438994    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"]
Nov 03 18:07:24.444878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.444850    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.488349 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.488314    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.488570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.488548    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qmf\" (UniqueName: \"kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf\") pod \"payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.504601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.504559    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"]
Nov 03 18:07:24.551027 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.550987    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c17c6a\""
Nov 03 18:07:24.551244 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.551186    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c17c6a-dockercfg-fv4vz\""
Nov 03 18:07:24.592985 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.592945    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.593193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.593152    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qmf\" (UniqueName: \"kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf\") pod \"payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.595871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.594566    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.661787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.659059    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qmf\" (UniqueName: \"kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf\") pod \"payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:24.780657 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:24.780566    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:25.172662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.172630    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"]
Nov 03 18:07:25.173115 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:07:25.173076    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5f15eb_80fb_4d53_afe9_9d3891cbe655.slice/crio-5b1800b47c93a29186d65a47f8cca6adaeb80e30ca899a0c96fc7e885862e117 WatchSource:0}: Error finding container 5b1800b47c93a29186d65a47f8cca6adaeb80e30ca899a0c96fc7e885862e117: Status 404 returned error can't find the container with id 5b1800b47c93a29186d65a47f8cca6adaeb80e30ca899a0c96fc7e885862e117
Nov 03 18:07:25.306579 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.306539    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7198-414c5a"]
Nov 03 18:07:25.311248 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.311220    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:25.366208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.366171    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6hkxc\""
Nov 03 18:07:25.368589 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.368559    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-414c5a"]
Nov 03 18:07:25.405882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.405837    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5kfv\" (UniqueName: \"kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv\") pod \"test-curl-7198-414c5a\" (UID: \"38945773-860d-403d-ada6-eb3d5944f1d0\") " pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:25.507276 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.507179    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5kfv\" (UniqueName: \"kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv\") pod \"test-curl-7198-414c5a\" (UID: \"38945773-860d-403d-ada6-eb3d5944f1d0\") " pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:25.572566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.572537    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5kfv\" (UniqueName: \"kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv\") pod \"test-curl-7198-414c5a\" (UID: \"38945773-860d-403d-ada6-eb3d5944f1d0\") " pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:25.624959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.624906    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266" event={"ID":"ab5f15eb-80fb-4d53-afe9-9d3891cbe655","Type":"ContainerStarted","Data":"a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e"}
Nov 03 18:07:25.624959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.624949    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:25.625218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.624987    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266" event={"ID":"ab5f15eb-80fb-4d53-afe9-9d3891cbe655","Type":"ContainerStarted","Data":"5b1800b47c93a29186d65a47f8cca6adaeb80e30ca899a0c96fc7e885862e117"}
Nov 03 18:07:25.625218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:25.625020    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:26.017404 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:26.017339    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266" podStartSLOduration=2.017316392 podStartE2EDuration="2.017316392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:25.747627294 +0000 UTC m=+1981.758365367" watchObservedRunningTime="2026-11-03 18:07:26.017316392 +0000 UTC m=+1982.028054465"
Nov 03 18:07:26.019182 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:07:26.019152    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38945773_860d_403d_ada6_eb3d5944f1d0.slice/crio-7ffab08d48318c4f3e5ddbc59a2209069c1083739e95975163fd7bb03c36c915 WatchSource:0}: Error finding container 7ffab08d48318c4f3e5ddbc59a2209069c1083739e95975163fd7bb03c36c915: Status 404 returned error can't find the container with id 7ffab08d48318c4f3e5ddbc59a2209069c1083739e95975163fd7bb03c36c915
Nov 03 18:07:26.019400 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:26.019374    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-414c5a"]
Nov 03 18:07:26.636842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:26.636801    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-414c5a" event={"ID":"38945773-860d-403d-ada6-eb3d5944f1d0","Type":"ContainerStarted","Data":"305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632"}
Nov 03 18:07:26.636842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:26.636847    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-414c5a" event={"ID":"38945773-860d-403d-ada6-eb3d5944f1d0","Type":"ContainerStarted","Data":"7ffab08d48318c4f3e5ddbc59a2209069c1083739e95975163fd7bb03c36c915"}
Nov 03 18:07:26.756372 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:26.756307    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7198-414c5a" podStartSLOduration=1.75629284 podStartE2EDuration="1.75629284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:26.753156439 +0000 UTC m=+1982.763894511" watchObservedRunningTime="2026-11-03 18:07:26.75629284 +0000 UTC m=+1982.767030911"
Nov 03 18:07:26.911675 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:26.911637    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:28.348292 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:28.348258    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7198-414c5a"]
Nov 03 18:07:28.354300 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:28.354266    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7198-414c5a"]
Nov 03 18:07:28.653260 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:28.653218    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7198-414c5a" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" containerName="test-curl-7198-414c5a" containerID="cri-o://305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632" gracePeriod=2
Nov 03 18:07:28.704984 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:07:28.704940    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-414c5a\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:30.926840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:30.926751    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:31.004338 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.004303    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv\" (UniqueName: \"kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv\") pod \"38945773-860d-403d-ada6-eb3d5944f1d0\" (UID: \"38945773-860d-403d-ada6-eb3d5944f1d0\") "
Nov 03 18:07:31.006899 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.006859    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv" pod "38945773-860d-403d-ada6-eb3d5944f1d0" (UID: "38945773-860d-403d-ada6-eb3d5944f1d0"). InnerVolumeSpecName "kube-api-access-f5kfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:31.106628 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.106589    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5kfv\" (UniqueName: \"kubernetes.io/projected/38945773-860d-403d-ada6-eb3d5944f1d0-kube-api-access-f5kfv\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:31.700760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.698321    5838 generic.go:356] "Generic (PLEG): container finished" podID="38945773-860d-403d-ada6-eb3d5944f1d0" containerID="305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632" exitCode=137
Nov 03 18:07:31.700760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.698644    5838 scope.go:122] "RemoveContainer" containerID="305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632"
Nov 03 18:07:31.700760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.698861    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:31.769701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.769662    5838 scope.go:122] "RemoveContainer" containerID="305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632"
Nov 03 18:07:31.780820 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:07:31.777526    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632\": container with ID starting with 305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632 not found: ID does not exist" containerID="305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632"
Nov 03 18:07:31.780820 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:31.777576    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632"} err="failed to get container status \"305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632\": rpc error: code = NotFound desc = could not find container \"305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632\": container with ID starting with 305b2f1527c1f9a09885ae3dd58450fa846cc0f4635a2fc6262f5a572b01d632 not found: ID does not exist"
Nov 03 18:07:31.795871 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:07:31.795829    5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-414c5a\" is forbidden: User \"system:node:ip-10-0-139-156.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-156.us-west-1.compute.internal' and this object" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" pod="opendatahub/test-curl-7198-414c5a"
Nov 03 18:07:32.121349 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:32.121252    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" path="/var/lib/kubelet/pods/38945773-860d-403d-ada6-eb3d5944f1d0/volumes"
Nov 03 18:07:32.572458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:32.572356    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:33.383464 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:33.383421    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:07:35.705757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:35.703854    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-82sdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:07:35.705757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:35.703940    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:36.463234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:36.463180    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:36.640807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:36.640776    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:07:36.974480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:36.974452    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:37.740343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:37.740304    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:38.118649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:38.118547    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:38.769856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:38.769822    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:40.347585 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:07:40.347541    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:07:41.269709 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:41.269674    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:41.423864 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:41.423829    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:42.007936 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:42.007896    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:42.184257 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:42.184223    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:42.553948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:42.553915    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:44.308333 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:44.308287    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:44.972515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:44.972418    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:45.305298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:45.305193    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:45.503922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:45.503880    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:46.061133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.061097    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"]
Nov 03 18:07:46.062306 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.062280    5838 cpu_manager.go:414] "containerMap: removing container" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" containerName="test-curl-7198-414c5a"
Nov 03 18:07:46.062306 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.062307    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" containerName="test-curl-7198-414c5a"
Nov 03 18:07:46.062528 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.062515    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="38945773-860d-403d-ada6-eb3d5944f1d0" containerName="test-curl-7198-414c5a"
Nov 03 18:07:46.069873 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.069842    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.127827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.127719    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"]
Nov 03 18:07:46.144009 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.143879    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mdn\" (UniqueName: \"kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn\") pod \"payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.144009 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.144014    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.156549 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.156517    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-528e53-dockercfg-8m7tb\""
Nov 03 18:07:46.172995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.172962    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-528e53\""
Nov 03 18:07:46.245935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.245895    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mdn\" (UniqueName: \"kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn\") pod \"payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.246135 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.246011    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.246816 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.246794    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.276597 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.276561    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"]
Nov 03 18:07:46.276820 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.276786    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" containerID="cri-o://c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497" gracePeriod=30
Nov 03 18:07:46.318423 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.318335    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mdn\" (UniqueName: \"kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn\") pod \"payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.393473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.393434    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:46.789705 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:07:46.789654    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5400cffc_6eb1_4f0a_9a60_eef0f0e27666.slice/crio-d799b7c246af6fa0d605614392a2adee9beba54f184d0d95ac11fcc1f9278904 WatchSource:0}: Error finding container d799b7c246af6fa0d605614392a2adee9beba54f184d0d95ac11fcc1f9278904: Status 404 returned error can't find the container with id d799b7c246af6fa0d605614392a2adee9beba54f184d0d95ac11fcc1f9278904
Nov 03 18:07:46.790148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:46.790123    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"]
Nov 03 18:07:47.076764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:47.076662    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx" event={"ID":"5400cffc-6eb1-4f0a-9a60-eef0f0e27666","Type":"ContainerStarted","Data":"ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c"}
Nov 03 18:07:47.076764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:47.076714    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx" event={"ID":"5400cffc-6eb1-4f0a-9a60-eef0f0e27666","Type":"ContainerStarted","Data":"d799b7c246af6fa0d605614392a2adee9beba54f184d0d95ac11fcc1f9278904"}
Nov 03 18:07:47.076953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:47.076773    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:07:47.199541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:47.199478    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx" podStartSLOduration=2.199456868 podStartE2EDuration="2.199456868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:47.198587843 +0000 UTC m=+2003.209325932" watchObservedRunningTime="2026-11-03 18:07:47.199456868 +0000 UTC m=+2003.210194948"
Nov 03 18:07:50.126833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:50.126790    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:50.888285 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:50.888245    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:07:50.888471 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:50.888306    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:07:51.560981 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.560892    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 18:07:51.631815 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.631776    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.631834    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.631916    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632042    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632111    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632284    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632356    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632399    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632452    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs\") pod \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\" (UID: \"1c318f8a-017b-4ca7-874d-4e95cc50fb56\") "
Nov 03 18:07:51.632699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632535    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:51.632953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.632923    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:51.633206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.633151    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:51.633477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.633266    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.633477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.633287    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.633477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.633324    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:51.633477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.633480    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:51.636628 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.636599    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "kube-api-access-chp8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:51.636766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.636704    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:51.636848 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.636784    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:07:51.636913 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.636850    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy" pod "1c318f8a-017b-4ca7-874d-4e95cc50fb56" (UID: "1c318f8a-017b-4ca7-874d-4e95cc50fb56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:51.734818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734777    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.734818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734811    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.734818 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734822    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.735113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734830    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.735113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734841    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chp8d\" (UniqueName: \"kubernetes.io/projected/1c318f8a-017b-4ca7-874d-4e95cc50fb56-kube-api-access-chp8d\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.735113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734852    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:51.735113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:51.734861    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c318f8a-017b-4ca7-874d-4e95cc50fb56-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:52.122951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.122917    5838 generic.go:356] "Generic (PLEG): container finished" podID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerID="c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497" exitCode=0
Nov 03 18:07:52.123246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.122973    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"
Nov 03 18:07:52.123246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.123003    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" event={"ID":"1c318f8a-017b-4ca7-874d-4e95cc50fb56","Type":"ContainerDied","Data":"c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497"}
Nov 03 18:07:52.123246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.123058    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm" event={"ID":"1c318f8a-017b-4ca7-874d-4e95cc50fb56","Type":"ContainerDied","Data":"561b8025f076930852578b43fb87e4c853b7af2483b6409c52d63fb4c338d413"}
Nov 03 18:07:52.123246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.123080    5838 scope.go:122] "RemoveContainer" containerID="c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497"
Nov 03 18:07:52.136574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.136550    5838 scope.go:122] "RemoveContainer" containerID="c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497"
Nov 03 18:07:52.136894 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:07:52.136873    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497\": container with ID starting with c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497 not found: ID does not exist" containerID="c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497"
Nov 03 18:07:52.136972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.136902    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497"} err="failed to get container status \"c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497\": rpc error: code = NotFound desc = could not find container \"c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497\": container with ID starting with c861de043c437ebc923889395d3fb775c8041cadad001a4cb11b53c065f74497 not found: ID does not exist"
Nov 03 18:07:52.296694 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.296658    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"]
Nov 03 18:07:52.346389 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:52.346353    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-gdnpm"]
Nov 03 18:07:54.120567 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:54.120529    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" path="/var/lib/kubelet/pods/1c318f8a-017b-4ca7-874d-4e95cc50fb56/volumes"
Nov 03 18:07:55.181938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:55.181886    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:58.093258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:07:58.093226    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:08:00.216891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:00.216844    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:05.164446 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.164405    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"]
Nov 03 18:08:05.165461 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.165436    5838 cpu_manager.go:414] "containerMap: removing container" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy"
Nov 03 18:08:05.165576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.165462    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy"
Nov 03 18:08:05.165711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.165696    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c318f8a-017b-4ca7-874d-4e95cc50fb56" containerName="istio-proxy"
Nov 03 18:08:05.171299 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.171271    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.231662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.231626    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"]
Nov 03 18:08:05.249342 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.249304    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.249627 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.249593    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.249799 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.249777    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7th\" (UniqueName: \"kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.249924 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.249895    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.275162 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.275112    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-2569c8-serving-cert\""
Nov 03 18:08:05.351509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.351469    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.351723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.351572    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7th\" (UniqueName: \"kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.351723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.351620    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.351723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.351664    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.351966 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:05.351882    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-serving-cert: secret "maas-api-e2e-shared-b-w6-2569c8-serving-cert" not found
Nov 03 18:08:05.351966 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:05.351944    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls podName:3baa4862-c70f-405b-a50f-b3c1664e249d nodeName:}" failed. No retries permitted until 2026-11-03 18:08:05.851927181 +0000 UTC m=+2021.862665249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls") pod "maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" (UID: "3baa4862-c70f-405b-a50f-b3c1664e249d") : secret "maas-api-e2e-shared-b-w6-2569c8-serving-cert" not found
Nov 03 18:08:05.352374 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.352350    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.352968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.352946    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:05.354491 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.354458    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.419772 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.419648    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7th\" (UniqueName: \"kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.538396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.538351    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:05.694510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.694439    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:05.860152 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.860108    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:05.862824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:05.862801    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:06.086453 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:06.086363    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:06.485894 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:06.485865    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"]
Nov 03 18:08:06.486299 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:08:06.486278    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3baa4862_c70f_405b_a50f_b3c1664e249d.slice/crio-6dd3f19fe0f51f28c3facd5a4f5e73a188e336df800e6bf7bb698b53082d9b0f WatchSource:0}: Error finding container 6dd3f19fe0f51f28c3facd5a4f5e73a188e336df800e6bf7bb698b53082d9b0f: Status 404 returned error can't find the container with id 6dd3f19fe0f51f28c3facd5a4f5e73a188e336df800e6bf7bb698b53082d9b0f
Nov 03 18:08:07.257230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:07.257193    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" event={"ID":"3baa4862-c70f-405b-a50f-b3c1664e249d","Type":"ContainerStarted","Data":"6dd3f19fe0f51f28c3facd5a4f5e73a188e336df800e6bf7bb698b53082d9b0f"}
Nov 03 18:08:08.266996 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:08.266959    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" event={"ID":"3baa4862-c70f-405b-a50f-b3c1664e249d","Type":"ContainerStarted","Data":"9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d"}
Nov 03 18:08:08.267421 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:08.267057    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:08.429053 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:08.428997    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" podStartSLOduration=2.593398708 podStartE2EDuration="3.428981521s" totalImagesPullingTime="835.582813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:08.427492837 +0000 UTC m=+2024.438230909" watchObservedRunningTime="2026-11-03 18:08:08.428981521 +0000 UTC m=+2024.439719593"
Nov 03 18:08:10.290619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.290574    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:10.366083 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:10.366034    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:08:10.643663 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.643626    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"]
Nov 03 18:08:10.649015 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.648993    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.707720 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.707683    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"]
Nov 03 18:08:10.734982 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.734945    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9hp\" (UniqueName: \"kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.734982 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.734994    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.735410 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.735387    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.735542 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.735422    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.752540 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.752512    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-19d275-serving-cert\""
Nov 03 18:08:10.836918 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.836879    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.836918 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.836928    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.837206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.837009    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9hp\" (UniqueName: \"kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.837206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.837045    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.837206 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:10.837061    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-serving-cert: secret "maas-api-e2e-shared-b-w2-19d275-serving-cert" not found
Nov 03 18:08:10.837206 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:10.837145    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls podName:b1bd643f-bde2-4d20-adc5-7e22da9306e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:08:11.337121767 +0000 UTC m=+2027.347859832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls") pod "maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" (UID: "b1bd643f-bde2-4d20-adc5-7e22da9306e9") : secret "maas-api-e2e-shared-b-w2-19d275-serving-cert" not found
Nov 03 18:08:10.837967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.837937    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.839197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.839169    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:10.840290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.840265    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:10.901778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:10.901670    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9hp\" (UniqueName: \"kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:11.008798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.008722    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:11.164756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.164631    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:11.329998 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.329964    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:11.348878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.348830    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:11.351604 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.351577    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:11.564242 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.564151    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:11.935784 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:11.934278    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:12.043061 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:08:12.040014    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bd643f_bde2_4d20_adc5_7e22da9306e9.slice/crio-6f7598492ab29beac32dad420cc4eabb7ed397b4039bfa93bd677ab02e58e561 WatchSource:0}: Error finding container 6f7598492ab29beac32dad420cc4eabb7ed397b4039bfa93bd677ab02e58e561: Status 404 returned error can't find the container with id 6f7598492ab29beac32dad420cc4eabb7ed397b4039bfa93bd677ab02e58e561
Nov 03 18:08:12.047762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:12.047514    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"]
Nov 03 18:08:12.192645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:12.192549    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:12.311481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:12.311437    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" event={"ID":"b1bd643f-bde2-4d20-adc5-7e22da9306e9","Type":"ContainerStarted","Data":"6f7598492ab29beac32dad420cc4eabb7ed397b4039bfa93bd677ab02e58e561"}
Nov 03 18:08:13.321975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:13.321931    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" event={"ID":"b1bd643f-bde2-4d20-adc5-7e22da9306e9","Type":"ContainerStarted","Data":"2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4"}
Nov 03 18:08:13.322410 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:13.322012    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:13.444459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:13.444401    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" podStartSLOduration=2.621165452 podStartE2EDuration="3.44437632s" totalImagesPullingTime="823.210868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:13.442647442 +0000 UTC m=+2029.453385576" watchObservedRunningTime="2026-11-03 18:08:13.44437632 +0000 UTC m=+2029.455114390"
Nov 03 18:08:14.281266 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:14.281237    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:08:16.469521 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:16.469482    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:17.323691 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:17.323656    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:17.522015 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:17.521975    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:17.692976 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:17.692945    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:19.335039 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:19.335010    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:08:20.648617 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:20.648574    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:20.853184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:20.853141    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:22.205312 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:22.205272    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:27.427245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:27.427204    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:29.768903 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:29.768867    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:31.595108 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:31.595075    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:31.778994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:31.778960    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:32.549786 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:32.549719    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:36.728585 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:36.728541    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:36.888339 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:36.888304    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:37.048758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:37.048651    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:37.861297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:37.861259    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:38.106417 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:38.106369    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:08:40.025140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:40.025100    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:40.224144 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:40.224054    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:40.355668 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:40.355576    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:08:42.442642 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:42.442517    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:43.808864 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:43.808826    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:48.638313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.638268    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"]
Nov 03 18:08:48.643523 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.643497    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.699148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.699109    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c17c6a-serving-cert\""
Nov 03 18:08:48.704210 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.704175    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"]
Nov 03 18:08:48.740664 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.740607    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545r6\" (UniqueName: \"kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.740927 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.740851    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.741138 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.741110    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.741279 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.741148    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.836374 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.836345    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:48.842834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.842803    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545r6\" (UniqueName: \"kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.842973 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.842908    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.843106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.843090    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.843106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.843120    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.843286 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:48.843206    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-serving-cert: secret "maas-api-e2e-inf-a-c17c6a-serving-cert" not found
Nov 03 18:08:48.843286 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:08:48.843272    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls podName:4036e36e-aa0a-4cf5-afda-77763c5a4dec nodeName:}" failed. No retries permitted until 2026-11-03 18:08:49.343257829 +0000 UTC m=+2065.353995878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls") pod "maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" (UID: "4036e36e-aa0a-4cf5-afda-77763c5a4dec") : secret "maas-api-e2e-inf-a-c17c6a-serving-cert" not found
Nov 03 18:08:48.843832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.843810    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.845880 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.845855    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.915109 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.913795    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-545r6\" (UniqueName: \"kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:48.997562 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:48.997525    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:49.182112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:49.182022    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:49.353459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:49.353423    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:49.356263 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:49.356229    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\") pod \"maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:49.558200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:49.558099    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:49.954056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:49.954013    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"]
Nov 03 18:08:49.955808 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:08:49.955778    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036e36e_aa0a_4cf5_afda_77763c5a4dec.slice/crio-829e9319792f5502bd4867c7c79b007befa6435fd13a4fc7023ea43331b84fd2 WatchSource:0}: Error finding container 829e9319792f5502bd4867c7c79b007befa6435fd13a4fc7023ea43331b84fd2: Status 404 returned error can't find the container with id 829e9319792f5502bd4867c7c79b007befa6435fd13a4fc7023ea43331b84fd2
Nov 03 18:08:49.957704 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:49.957683    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:08:50.699142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:50.699099    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" event={"ID":"4036e36e-aa0a-4cf5-afda-77763c5a4dec","Type":"ContainerStarted","Data":"829e9319792f5502bd4867c7c79b007befa6435fd13a4fc7023ea43331b84fd2"}
Nov 03 18:08:51.709596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:51.709557    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" event={"ID":"4036e36e-aa0a-4cf5-afda-77763c5a4dec","Type":"ContainerStarted","Data":"2ff935ba86b553eae8293029ef455c962a4ea1c64d48397018948dff34da164d"}
Nov 03 18:08:51.710109 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:51.709622    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:51.832824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:51.832765    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" podStartSLOduration=3.001541975 podStartE2EDuration="3.832714585s" totalImagesPullingTime="831.17261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:51.831854582 +0000 UTC m=+2067.842592688" watchObservedRunningTime="2026-11-03 18:08:51.832714585 +0000 UTC m=+2067.843452676"
Nov 03 18:08:54.336356 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:54.336301    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:54.493111 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:54.493074    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:57.725183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:57.725151    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:08:59.500605 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:08:59.500562    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:04.611724 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:04.611639    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:09.705685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:09.705641    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:10.344235 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:10.344195    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:09:14.545163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.545126    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"]
Nov 03 18:09:14.550410 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.550380    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.639030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.638986    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.639234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.639051    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjwv\" (UniqueName: \"kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.639234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.639181    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.639335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.639230    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.655363 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.655319    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-528e53-serving-cert\""
Nov 03 18:09:14.659887 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.659856    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"]
Nov 03 18:09:14.741237 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.741199    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.741450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.741316    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjwv\" (UniqueName: \"kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.741450 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:14.741363    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-serving-cert: secret "maas-api-e2e-shared-b-w4-528e53-serving-cert" not found
Nov 03 18:09:14.741450 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:14.741437    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls podName:bc7e6943-1d9a-49c3-a89f-91110e6093c2 nodeName:}" failed. No retries permitted until 2026-11-03 18:09:15.241415892 +0000 UTC m=+2091.252153946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls") pod "maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" (UID: "bc7e6943-1d9a-49c3-a89f-91110e6093c2") : secret "maas-api-e2e-shared-b-w4-528e53-serving-cert" not found
Nov 03 18:09:14.741626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.741459    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.741626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.741483    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:14.741626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.741529    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.742272 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.742246    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.744251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.744228    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.812177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.811898    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjwv\" (UniqueName: \"kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:14.909624 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:14.909588    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:15.106487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:15.106388    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:15.258934 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:15.258898    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:15.261504 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:15.261478    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:15.467461 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:15.467424    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:15.865816 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:15.865780    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"]
Nov 03 18:09:15.866912 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:09:15.866879    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7e6943_1d9a_49c3_a89f_91110e6093c2.slice/crio-48f9a7933dd6e600479c2e29651c8db939babfc7f26b71229da4f602e3ce5ab0 WatchSource:0}: Error finding container 48f9a7933dd6e600479c2e29651c8db939babfc7f26b71229da4f602e3ce5ab0: Status 404 returned error can't find the container with id 48f9a7933dd6e600479c2e29651c8db939babfc7f26b71229da4f602e3ce5ab0
Nov 03 18:09:15.937563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:15.937521    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" event={"ID":"bc7e6943-1d9a-49c3-a89f-91110e6093c2","Type":"ContainerStarted","Data":"48f9a7933dd6e600479c2e29651c8db939babfc7f26b71229da4f602e3ce5ab0"}
Nov 03 18:09:16.949000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:16.948952    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" event={"ID":"bc7e6943-1d9a-49c3-a89f-91110e6093c2","Type":"ContainerStarted","Data":"c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f"}
Nov 03 18:09:16.949409 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:16.949043    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:17.075317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:17.075204    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" podStartSLOduration=2.166203641 podStartE2EDuration="3.075189221s" totalImagesPullingTime="908.98558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:17.071806702 +0000 UTC m=+2093.082544774" watchObservedRunningTime="2026-11-03 18:09:17.075189221 +0000 UTC m=+2093.085927292"
Nov 03 18:09:18.009619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:18.009575    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:19.996241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:19.996205    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:22.962683 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:22.962641    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:09:24.588606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:24.588566    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:09:24.592991 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:24.592963    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:09:24.989577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:24.989539    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:29.952219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:29.952185    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:35.017843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:35.017785    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:40.326759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:40.325996    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:40.364954 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:40.364918    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:09:42.815026 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:42.814981    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:44.696249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:44.696205    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"]
Nov 03 18:09:44.696804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:44.696519    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy" containerID="cri-o://4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24" gracePeriod=30
Nov 03 18:09:44.747830 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:44.747779    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"]
Nov 03 18:09:44.748030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:44.747993    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" containerID="cri-o://1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2" gracePeriod=30
Nov 03 18:09:44.839942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:44.839903    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"]
Nov 03 18:09:44.840145 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:44.840123    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy" containerID="cri-o://72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e" gracePeriod=30
Nov 03 18:09:45.940326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:45.940288    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:46.038899 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:46.038858    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"]
Nov 03 18:09:46.039384 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:46.039357    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" containerID="cri-o://24b57541fe10b24a36e02a941c71f1dc8beb6a34d6bcd136f2f45c5fde6b8e91" gracePeriod=30
Nov 03 18:09:46.166424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:46.166386    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:46.166702 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:46.166445    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:47.158837 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:47.158804    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"]
Nov 03 18:09:47.159310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:47.159037    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" containerID="cri-o://8eff44e4f7b968059ae42c75180296fd169f1d8cb2ab802d1450095ea3d2bc31" gracePeriod=30
Nov 03 18:09:47.814725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:47.814684    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:09:48.100179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:48.100088    5838 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:48.100179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:48.100143    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:49.774680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:49.774644    5838 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:49.775191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:49.774715    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:50.068066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.068028    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 18:09:50.113488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113444    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113528    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113567    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113594    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113640    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113698    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113804    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.113921    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.114123    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.114539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.114409    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:50.115151 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.114805    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.118453 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.117885    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.119475 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.119444    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.119599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.119555    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.119599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.119593    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token\") pod \"cee56aec-eddf-4726-8ab6-b3a71133cf69\" (UID: \"cee56aec-eddf-4726-8ab6-b3a71133cf69\") "
Nov 03 18:09:50.120428 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.120392    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.120524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.120435    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.120524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.120450    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.120524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.120465    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.120524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.120478    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56aec-eddf-4726-8ab6-b3a71133cf69-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.120524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.120491    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56aec-eddf-4726-8ab6-b3a71133cf69-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.125126 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.124976    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.126056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.126027    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "kube-api-access-l7trj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.128650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.128619    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo" pod "cee56aec-eddf-4726-8ab6-b3a71133cf69" (UID: "cee56aec-eddf-4726-8ab6-b3a71133cf69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:50.136030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.136003    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 18:09:50.172834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.172809    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 18:09:50.176800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.176770    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"]
Nov 03 18:09:50.221504 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221410    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.221504 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221472    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.221504 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221502    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221525    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221557    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221582    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221602    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221637    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221678    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221724    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.221865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221799    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221901    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221915    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221897    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.221926    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222039    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222077    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222156    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222200    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo\") pod \"6c797500-276e-4849-a923-c6fc153bafce\" (UID: \"6c797500-276e-4849-a923-c6fc153bafce\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222226    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222264    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm\") pod \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\" (UID: \"41f6489f-0097-4ff1-9658-3e8a90edf1c6\") "
Nov 03 18:09:50.222450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222273    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:50.223065 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.222817    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:50.223126 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223058    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.223377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223353    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223382    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223397    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223416    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223431    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223446    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223459    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7trj\" (UniqueName: \"kubernetes.io/projected/cee56aec-eddf-4726-8ab6-b3a71133cf69-kube-api-access-l7trj\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223480    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c797500-276e-4849-a923-c6fc153bafce-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223495    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56aec-eddf-4726-8ab6-b3a71133cf69-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.223760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223722    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.223949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.223930    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.224144 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.224121    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.224344 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.224324    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.224588 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.224561    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.224663 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.224578    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:50.224663 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.224561    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.225012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.224989    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.225282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.225260    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.225573 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.225554    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "kube-api-access-k6wsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.226310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.226293    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm" pod "41f6489f-0097-4ff1-9658-3e8a90edf1c6" (UID: "41f6489f-0097-4ff1-9658-3e8a90edf1c6"). InnerVolumeSpecName "kube-api-access-4v6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.226482 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.226463    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo" pod "6c797500-276e-4849-a923-c6fc153bafce" (UID: "6c797500-276e-4849-a923-c6fc153bafce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:50.255106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.255067    5838 generic.go:356] "Generic (PLEG): container finished" podID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerID="1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2" exitCode=0
Nov 03 18:09:50.255289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.255137    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"
Nov 03 18:09:50.255289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.255170    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" event={"ID":"41f6489f-0097-4ff1-9658-3e8a90edf1c6","Type":"ContainerDied","Data":"1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2"}
Nov 03 18:09:50.255289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.255234    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk" event={"ID":"41f6489f-0097-4ff1-9658-3e8a90edf1c6","Type":"ContainerDied","Data":"10fdbbe665d8144a2350a3b44028844a95d0f898be90843fc679906afbc6da33"}
Nov 03 18:09:50.255289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.255262    5838 scope.go:122] "RemoveContainer" containerID="1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2"
Nov 03 18:09:50.257260 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.257230    5838 generic.go:356] "Generic (PLEG): container finished" podID="6c797500-276e-4849-a923-c6fc153bafce" containerID="72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e" exitCode=0
Nov 03 18:09:50.257372 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.257326    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" event={"ID":"6c797500-276e-4849-a923-c6fc153bafce","Type":"ContainerDied","Data":"72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e"}
Nov 03 18:09:50.257439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.257366    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9" event={"ID":"6c797500-276e-4849-a923-c6fc153bafce","Type":"ContainerDied","Data":"c86e93cf48f99a78c2ccb510f80cfbaa09d346756628cab511359955f4627f1f"}
Nov 03 18:09:50.257439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.257429    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"
Nov 03 18:09:50.264633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.263454    5838 generic.go:356] "Generic (PLEG): container finished" podID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerID="4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24" exitCode=0
Nov 03 18:09:50.264633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.263689    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" event={"ID":"cee56aec-eddf-4726-8ab6-b3a71133cf69","Type":"ContainerDied","Data":"4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24"}
Nov 03 18:09:50.264633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.263759    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72" event={"ID":"cee56aec-eddf-4726-8ab6-b3a71133cf69","Type":"ContainerDied","Data":"294beee60fd30c85d9aeff98119c08f19519f4c4f7f938f25176fef0b3718c99"}
Nov 03 18:09:50.264633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.263838    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"
Nov 03 18:09:50.282241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.282218    5838 scope.go:122] "RemoveContainer" containerID="1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2"
Nov 03 18:09:50.282647 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:50.282620    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2\": container with ID starting with 1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2 not found: ID does not exist" containerID="1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2"
Nov 03 18:09:50.282838 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.282657    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2"} err="failed to get container status \"1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2\": rpc error: code = NotFound desc = could not find container \"1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2\": container with ID starting with 1ad06500cf9e71c2400da0a86bf320553ee8fc7479580eb27d5e96937fdb39f2 not found: ID does not exist"
Nov 03 18:09:50.282838 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.282675    5838 scope.go:122] "RemoveContainer" containerID="72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e"
Nov 03 18:09:50.294897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.294874    5838 scope.go:122] "RemoveContainer" containerID="72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e"
Nov 03 18:09:50.295263 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:50.295228    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e\": container with ID starting with 72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e not found: ID does not exist" containerID="72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e"
Nov 03 18:09:50.295358 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.295273    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e"} err="failed to get container status \"72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e\": rpc error: code = NotFound desc = could not find container \"72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e\": container with ID starting with 72678f84b383b3a727b88ae390a63fa1db03d178b6473ceeb512af5f938bfe7e not found: ID does not exist"
Nov 03 18:09:50.295358 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.295299    5838 scope.go:122] "RemoveContainer" containerID="4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24"
Nov 03 18:09:50.299004 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.298980    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"]
Nov 03 18:09:50.307168 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.307137    5838 scope.go:122] "RemoveContainer" containerID="4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24"
Nov 03 18:09:50.307474 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:50.307450    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24\": container with ID starting with 4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24 not found: ID does not exist" containerID="4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24"
Nov 03 18:09:50.307530 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.307484    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24"} err="failed to get container status \"4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24\": rpc error: code = NotFound desc = could not find container \"4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24\": container with ID starting with 4234c9713fd43dd7c139a89310894e5e35e48e7364a9a19dd54c75f1f3bc1e24 not found: ID does not exist"
Nov 03 18:09:50.325207 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325175    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325207 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325206    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325216    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325225    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c797500-276e-4849-a923-c6fc153bafce-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325235    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325244    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6zm\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-kube-api-access-4v6zm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325253    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325261    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325269    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c797500-276e-4849-a923-c6fc153bafce-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325276    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325284    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/6c797500-276e-4849-a923-c6fc153bafce-kube-api-access-k6wsd\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.325424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.325292    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f6489f-0097-4ff1-9658-3e8a90edf1c6-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.519791 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.519636    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:50.677012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.676976    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:50.771209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.771112    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"]
Nov 03 18:09:50.771209 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.771180    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-mmktk"]
Nov 03 18:09:50.882377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.882340    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"]
Nov 03 18:09:50.888502 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:50.888469    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-w92q9"]
Nov 03 18:09:51.135266 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.135232    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"]
Nov 03 18:09:51.139867 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.139837    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-wrm72"]
Nov 03 18:09:51.278886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.278845    5838 generic.go:356] "Generic (PLEG): container finished" podID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerID="24b57541fe10b24a36e02a941c71f1dc8beb6a34d6bcd136f2f45c5fde6b8e91" exitCode=0
Nov 03 18:09:51.279089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.278920    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" event={"ID":"8321e086-def0-4963-9ef0-f7e5c180f7f5","Type":"ContainerDied","Data":"24b57541fe10b24a36e02a941c71f1dc8beb6a34d6bcd136f2f45c5fde6b8e91"}
Nov 03 18:09:51.331609 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.331581    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 18:09:51.464030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.463991    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464186 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464075    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464186 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464106    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464197    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464255    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:51.464435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464346    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464494 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464476    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464494 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464475    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:51.464584 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464538    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464609    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464676 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464652    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token\") pod \"8321e086-def0-4963-9ef0-f7e5c180f7f5\" (UID: \"8321e086-def0-4963-9ef0-f7e5c180f7f5\") "
Nov 03 18:09:51.464776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464718    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:51.464906 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.464876    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:51.465034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.465003    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:51.465482 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.465454    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.465578 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.465485    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.465578 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.465500    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8321e086-def0-4963-9ef0-f7e5c180f7f5-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.465578 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.465515    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.465578 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.465528    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.467685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.467658    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:51.467685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.467677    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:51.470902 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.470879    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "kube-api-access-8fm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:51.471011 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.470911    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token" pod "8321e086-def0-4963-9ef0-f7e5c180f7f5" (UID: "8321e086-def0-4963-9ef0-f7e5c180f7f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:51.566581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.566485    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.566581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.566519    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.566581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.566532    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8321e086-def0-4963-9ef0-f7e5c180f7f5-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.566581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.566545    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fm2l\" (UniqueName: \"kubernetes.io/projected/8321e086-def0-4963-9ef0-f7e5c180f7f5-kube-api-access-8fm2l\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.848023 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.847927    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"]
Nov 03 18:09:51.849500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849389    5838 cpu_manager.go:414] "containerMap: removing container" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy"
Nov 03 18:09:51.849500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849416    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy"
Nov 03 18:09:51.849500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849473    5838 cpu_manager.go:414] "containerMap: removing container" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy"
Nov 03 18:09:51.849500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849481    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy"
Nov 03 18:09:51.849794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849509    5838 cpu_manager.go:414] "containerMap: removing container" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy"
Nov 03 18:09:51.849794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849519    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy"
Nov 03 18:09:51.849794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849560    5838 cpu_manager.go:414] "containerMap: removing container" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy"
Nov 03 18:09:51.849794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849567    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy"
Nov 03 18:09:51.849794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849785    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" containerName="istio-proxy"
Nov 03 18:09:51.849992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849811    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" containerName="istio-proxy"
Nov 03 18:09:51.849992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849825    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c797500-276e-4849-a923-c6fc153bafce" containerName="istio-proxy"
Nov 03 18:09:51.849992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.849839    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" containerName="istio-proxy"
Nov 03 18:09:51.855068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.855044    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:51.910339 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.910293    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-640dc9-serving-cert\""
Nov 03 18:09:51.912243 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.912206    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"]
Nov 03 18:09:51.973838 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.973795    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:51.974061 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.973878    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:51.974061 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.973996    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:51.974173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:51.974066    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmm4\" (UniqueName: \"kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.072331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.072290    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:52.075782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.075755    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.075900 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.075884    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.075981 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.075962    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmm4\" (UniqueName: \"kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.076047 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:52.076032    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-serving-cert: secret "maas-api-e2e-api-b-640dc9-serving-cert" not found
Nov 03 18:09:52.076101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.076085    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.076153 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:52.076106    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls podName:f2b5f5de-cb0f-440f-98d1-fead0c207e59 nodeName:}" failed. No retries permitted until 2026-11-03 18:09:52.576085693 +0000 UTC m=+2128.586823763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls") pod "maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" (UID: "f2b5f5de-cb0f-440f-98d1-fead0c207e59") : secret "maas-api-e2e-api-b-640dc9-serving-cert" not found
Nov 03 18:09:52.076697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.076677    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.078513 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.078484    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.128208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.128109    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f6489f-0097-4ff1-9658-3e8a90edf1c6" path="/var/lib/kubelet/pods/41f6489f-0097-4ff1-9658-3e8a90edf1c6/volumes"
Nov 03 18:09:52.129887 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.128900    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c797500-276e-4849-a923-c6fc153bafce" path="/var/lib/kubelet/pods/6c797500-276e-4849-a923-c6fc153bafce/volumes"
Nov 03 18:09:52.129887 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.129644    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee56aec-eddf-4726-8ab6-b3a71133cf69" path="/var/lib/kubelet/pods/cee56aec-eddf-4726-8ab6-b3a71133cf69/volumes"
Nov 03 18:09:52.143796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.143025    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmm4\" (UniqueName: \"kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.278340 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.278306    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:52.296122 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.296090    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"
Nov 03 18:09:52.296298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.296091    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968" event={"ID":"8321e086-def0-4963-9ef0-f7e5c180f7f5","Type":"ContainerDied","Data":"5df3302f6e1dd8bd4453ea0621608d347050210434b7a93eaab6bb2c89da20dc"}
Nov 03 18:09:52.296298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.296239    5838 scope.go:122] "RemoveContainer" containerID="24b57541fe10b24a36e02a941c71f1dc8beb6a34d6bcd136f2f45c5fde6b8e91"
Nov 03 18:09:52.298392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.298358    5838 generic.go:356] "Generic (PLEG): container finished" podID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerID="8eff44e4f7b968059ae42c75180296fd169f1d8cb2ab802d1450095ea3d2bc31" exitCode=0
Nov 03 18:09:52.298506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.298420    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" event={"ID":"b222a6c8-acbb-42a0-8a33-e5056c50458d","Type":"ContainerDied","Data":"8eff44e4f7b968059ae42c75180296fd169f1d8cb2ab802d1450095ea3d2bc31"}
Nov 03 18:09:52.430142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.430099    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:52.471662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.471630    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"]
Nov 03 18:09:52.480871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.480833    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-92968"]
Nov 03 18:09:52.482184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.482166    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 18:09:52.604477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604440    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.604759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604482    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.604759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604546    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.604759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604600    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.604759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604709    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.605042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604814    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.605042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604815    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:52.605042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.604959    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.605196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.605049    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:52.605196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.605063    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.605196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.605185    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs\") pod \"b222a6c8-acbb-42a0-8a33-e5056c50458d\" (UID: \"b222a6c8-acbb-42a0-8a33-e5056c50458d\") "
Nov 03 18:09:52.605418 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.605237    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:52.606813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.606711    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:52.607221 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.607155    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.607581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.607554    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.607795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.607583    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.607795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.607599    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.607795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.607612    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b222a6c8-acbb-42a0-8a33-e5056c50458d-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.607795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.607700    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:52.608293 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.608263    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:52.608863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.608834    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:52.608968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.608935    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:52.610064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.610039    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s" pod "b222a6c8-acbb-42a0-8a33-e5056c50458d" (UID: "b222a6c8-acbb-42a0-8a33-e5056c50458d"). InnerVolumeSpecName "kube-api-access-7k88s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:52.610268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.610246    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\") pod \"maas-api-e2e-api-b-640dc9-556489c96d-tpxb4\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:52.708645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.708557    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.708645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.708589    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.708645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.708599    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b222a6c8-acbb-42a0-8a33-e5056c50458d-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.708645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.708608    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b222a6c8-acbb-42a0-8a33-e5056c50458d-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.708645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.708617    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k88s\" (UniqueName: \"kubernetes.io/projected/b222a6c8-acbb-42a0-8a33-e5056c50458d-kube-api-access-7k88s\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.782147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:52.782108    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:53.183098 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.183067    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"]
Nov 03 18:09:53.183539 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:09:53.183510    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b5f5de_cb0f_440f_98d1_fead0c207e59.slice/crio-feb439f269b1676b1112ca968f0f1e846e5bc55ccf7268b39a25349c79768efa WatchSource:0}: Error finding container feb439f269b1676b1112ca968f0f1e846e5bc55ccf7268b39a25349c79768efa: Status 404 returned error can't find the container with id feb439f269b1676b1112ca968f0f1e846e5bc55ccf7268b39a25349c79768efa
Nov 03 18:09:53.309652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.309625    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"
Nov 03 18:09:53.309900 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.309660    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr" event={"ID":"b222a6c8-acbb-42a0-8a33-e5056c50458d","Type":"ContainerDied","Data":"22624fd3c654cfbe7b7a1a406056ccb3cf6fa64bd8a39d3ba2d50425b364edda"}
Nov 03 18:09:53.309900 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.309726    5838 scope.go:122] "RemoveContainer" containerID="8eff44e4f7b968059ae42c75180296fd169f1d8cb2ab802d1450095ea3d2bc31"
Nov 03 18:09:53.311348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.311324    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" event={"ID":"f2b5f5de-cb0f-440f-98d1-fead0c207e59","Type":"ContainerStarted","Data":"feb439f269b1676b1112ca968f0f1e846e5bc55ccf7268b39a25349c79768efa"}
Nov 03 18:09:53.482949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.482873    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"]
Nov 03 18:09:53.534020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:53.533969    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-lmmbr"]
Nov 03 18:09:54.126258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:54.126205    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8321e086-def0-4963-9ef0-f7e5c180f7f5" path="/var/lib/kubelet/pods/8321e086-def0-4963-9ef0-f7e5c180f7f5/volumes"
Nov 03 18:09:54.126891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:54.126871    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" path="/var/lib/kubelet/pods/b222a6c8-acbb-42a0-8a33-e5056c50458d/volumes"
Nov 03 18:09:54.322968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:54.322874    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" event={"ID":"f2b5f5de-cb0f-440f-98d1-fead0c207e59","Type":"ContainerStarted","Data":"685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9"}
Nov 03 18:09:54.322968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:54.322954    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:09:54.445896 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:54.445841    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" podStartSLOduration=2.583614177 podStartE2EDuration="3.445824187s" totalImagesPullingTime="862.21001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:54.443774022 +0000 UTC m=+2130.454512097" watchObservedRunningTime="2026-11-03 18:09:54.445824187 +0000 UTC m=+2130.456562256"
Nov 03 18:09:55.177125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.177077    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz" podUID="7268918d-bd86-4dc9-9ce0-649523e28cf4" containerName="payload-processing" containerID="cri-o://78ed7f4d0fe12eed90534fee31b8b0e44f719c29bf3891b393d268a9fec254f9" gracePeriod=25
Nov 03 18:09:55.299456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.299389    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj" podUID="133954e9-f4c9-4c11-9c4b-2e196572e9a8" containerName="payload-pre-processing" containerID="cri-o://7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db" gracePeriod=25
Nov 03 18:09:55.337359 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.337316    5838 generic.go:356] "Generic (PLEG): container finished" podID="7268918d-bd86-4dc9-9ce0-649523e28cf4" containerID="78ed7f4d0fe12eed90534fee31b8b0e44f719c29bf3891b393d268a9fec254f9" exitCode=0
Nov 03 18:09:55.337849 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.337390    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz" event={"ID":"7268918d-bd86-4dc9-9ce0-649523e28cf4","Type":"ContainerDied","Data":"78ed7f4d0fe12eed90534fee31b8b0e44f719c29bf3891b393d268a9fec254f9"}
Nov 03 18:09:55.528133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.528101    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:09:55.635105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.635073    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:09:55.652135 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.652084    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume\") pod \"7268918d-bd86-4dc9-9ce0-649523e28cf4\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") "
Nov 03 18:09:55.652342 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.652161    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk\" (UniqueName: \"kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk\") pod \"7268918d-bd86-4dc9-9ce0-649523e28cf4\" (UID: \"7268918d-bd86-4dc9-9ce0-649523e28cf4\") "
Nov 03 18:09:55.653233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.653200    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume" pod "7268918d-bd86-4dc9-9ce0-649523e28cf4" (UID: "7268918d-bd86-4dc9-9ce0-649523e28cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:55.655793 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.655759    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk" pod "7268918d-bd86-4dc9-9ce0-649523e28cf4" (UID: "7268918d-bd86-4dc9-9ce0-649523e28cf4"). InnerVolumeSpecName "kube-api-access-jsnhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:55.758935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.758637    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28\" (UniqueName: \"kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28\") pod \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") "
Nov 03 18:09:55.758935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.758868    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume\") pod \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\" (UID: \"133954e9-f4c9-4c11-9c4b-2e196572e9a8\") "
Nov 03 18:09:55.764172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.764018    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume" pod "133954e9-f4c9-4c11-9c4b-2e196572e9a8" (UID: "133954e9-f4c9-4c11-9c4b-2e196572e9a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:55.764172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.764117    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsnhk\" (UniqueName: \"kubernetes.io/projected/7268918d-bd86-4dc9-9ce0-649523e28cf4-kube-api-access-jsnhk\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:55.764172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.764146    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7268918d-bd86-4dc9-9ce0-649523e28cf4-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:55.774757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.766648    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28" pod "133954e9-f4c9-4c11-9c4b-2e196572e9a8" (UID: "133954e9-f4c9-4c11-9c4b-2e196572e9a8"). InnerVolumeSpecName "kube-api-access-b9l28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:55.866072 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.866033    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9l28\" (UniqueName: \"kubernetes.io/projected/133954e9-f4c9-4c11-9c4b-2e196572e9a8-kube-api-access-b9l28\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:55.866072 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:55.866073    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133954e9-f4c9-4c11-9c4b-2e196572e9a8-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:56.349108 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.349075    5838 generic.go:356] "Generic (PLEG): container finished" podID="133954e9-f4c9-4c11-9c4b-2e196572e9a8" containerID="7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db" exitCode=0
Nov 03 18:09:56.349629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.349132    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"
Nov 03 18:09:56.349629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.349164    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj" event={"ID":"133954e9-f4c9-4c11-9c4b-2e196572e9a8","Type":"ContainerDied","Data":"7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db"}
Nov 03 18:09:56.349629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.349224    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj" event={"ID":"133954e9-f4c9-4c11-9c4b-2e196572e9a8","Type":"ContainerDied","Data":"9374908e8cb496c5ab94f6e75ea70ede8ed4406e40908fe6acc8059ca89caa87"}
Nov 03 18:09:56.349629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.349267    5838 scope.go:122] "RemoveContainer" containerID="7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db"
Nov 03 18:09:56.351636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.351587    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz" event={"ID":"7268918d-bd86-4dc9-9ce0-649523e28cf4","Type":"ContainerDied","Data":"c628f47780a6cd53ab81c951722dfe5d2e3e937ae779edd893f238575d2b03bf"}
Nov 03 18:09:56.351775 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.351714    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"
Nov 03 18:09:56.361524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.361499    5838 scope.go:122] "RemoveContainer" containerID="7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db"
Nov 03 18:09:56.361904 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:09:56.361882    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db\": container with ID starting with 7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db not found: ID does not exist" containerID="7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db"
Nov 03 18:09:56.362002 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.361919    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db"} err="failed to get container status \"7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db\": rpc error: code = NotFound desc = could not find container \"7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db\": container with ID starting with 7dcb681b3123fe7c7adda68339b29717ce4f84574379800553a10315b0c428db not found: ID does not exist"
Nov 03 18:09:56.362002 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.361946    5838 scope.go:122] "RemoveContainer" containerID="78ed7f4d0fe12eed90534fee31b8b0e44f719c29bf3891b393d268a9fec254f9"
Nov 03 18:09:56.521717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.521670    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"]
Nov 03 18:09:56.569466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.569422    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ad914d-57c9c6979-twllj"]
Nov 03 18:09:56.642149 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.642117    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"]
Nov 03 18:09:56.648443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:56.648409    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ad914d-6898c75467-dhjmz"]
Nov 03 18:09:57.782181 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:57.782144    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:57.939639 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:57.939607    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:58.119298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:58.119208    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133954e9-f4c9-4c11-9c4b-2e196572e9a8" path="/var/lib/kubelet/pods/133954e9-f4c9-4c11-9c4b-2e196572e9a8/volumes"
Nov 03 18:09:58.119702 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:58.119684    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7268918d-bd86-4dc9-9ce0-649523e28cf4" path="/var/lib/kubelet/pods/7268918d-bd86-4dc9-9ce0-649523e28cf4/volumes"
Nov 03 18:09:58.569322 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:09:58.569290    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:00.343846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:00.343809    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:10:04.028121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:04.028076    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:09.231254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:09.231210    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:10.342874 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:10:10.342830    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:10:11.270554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:11.270515    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:11.427391 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:11.427347    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:14.806694 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:14.806653    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:19.545721 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.545680    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"]
Nov 03 18:10:19.546531 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546506    5838 cpu_manager.go:414] "containerMap: removing container" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy"
Nov 03 18:10:19.546531 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546528    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy"
Nov 03 18:10:19.546699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546550    5838 cpu_manager.go:414] "containerMap: removing container" podUID="7268918d-bd86-4dc9-9ce0-649523e28cf4" containerName="payload-processing"
Nov 03 18:10:19.546699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546556    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7268918d-bd86-4dc9-9ce0-649523e28cf4" containerName="payload-processing"
Nov 03 18:10:19.546699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546566    5838 cpu_manager.go:414] "containerMap: removing container" podUID="133954e9-f4c9-4c11-9c4b-2e196572e9a8" containerName="payload-pre-processing"
Nov 03 18:10:19.546699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546574    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133954e9-f4c9-4c11-9c4b-2e196572e9a8" containerName="payload-pre-processing"
Nov 03 18:10:19.546850 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546744    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b222a6c8-acbb-42a0-8a33-e5056c50458d" containerName="istio-proxy"
Nov 03 18:10:19.546850 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546757    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7268918d-bd86-4dc9-9ce0-649523e28cf4" containerName="payload-processing"
Nov 03 18:10:19.546850 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.546765    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="133954e9-f4c9-4c11-9c4b-2e196572e9a8" containerName="payload-pre-processing"
Nov 03 18:10:19.550764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.550725    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.613091 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.613049    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"]
Nov 03 18:10:19.655569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.655533    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.655829 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.655644    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.655829 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.655818    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c8j\" (UniqueName: \"kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.655990 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.655897    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.656870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.656849    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f5100c-serving-cert\""
Nov 03 18:10:19.744569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.744527    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:19.757151 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.757112    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c8j\" (UniqueName: \"kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.757343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.757195    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.757343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.757295    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.757343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.757329    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.758004 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.757980    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.759881 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.759860    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.759994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.759918    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.821272 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.821192    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c8j\" (UniqueName: \"kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j\") pod \"maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.866536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.866501    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:19.899057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:19.899022    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:20.052558 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:20.052421    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:20.281034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:20.281006    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"]
Nov 03 18:10:20.281816 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:10:20.281786    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e64bcbf_4f4d_407e_981e_5d36b10ff380.slice/crio-64798d4b0178ce2fdb3ddea6eb72a7126c2bd1990b79d718027a6bb1533d6789 WatchSource:0}: Error finding container 64798d4b0178ce2fdb3ddea6eb72a7126c2bd1990b79d718027a6bb1533d6789: Status 404 returned error can't find the container with id 64798d4b0178ce2fdb3ddea6eb72a7126c2bd1990b79d718027a6bb1533d6789
Nov 03 18:10:20.565395 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:20.565308    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4" event={"ID":"9e64bcbf-4f4d-407e-981e-5d36b10ff380","Type":"ContainerStarted","Data":"64798d4b0178ce2fdb3ddea6eb72a7126c2bd1990b79d718027a6bb1533d6789"}
Nov 03 18:10:21.575450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:21.575363    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4" event={"ID":"9e64bcbf-4f4d-407e-981e-5d36b10ff380","Type":"ContainerStarted","Data":"406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e"}
Nov 03 18:10:21.575450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:21.575432    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:21.697941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:21.697888    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4" podStartSLOduration=1.7598869129999999 podStartE2EDuration="2.697871129s" totalImagesPullingTime="937.984216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:21.696150344 +0000 UTC m=+2157.706888435" watchObservedRunningTime="2026-11-03 18:10:21.697871129 +0000 UTC m=+2157.708609201"
Nov 03 18:10:25.528000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:25.527950    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:27.589175 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:27.589139    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:10:30.024617 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:30.024579    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:30.181814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:30.181776    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:30.793034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:30.792997    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:31.796655 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:31.796608    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"]
Nov 03 18:10:32.156268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:32.156216    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:32.311697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:32.311665    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:33.523557 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:33.523508    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:33.680324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:33.677335    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:35.193344 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.193311    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"]
Nov 03 18:10:35.193804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.193512    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" podUID="3a79e47c-937a-4adb-b3db-fcb4c900aebf" containerName="maas-api" containerID="cri-o://b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde" gracePeriod=30
Nov 03 18:10:35.292077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.292044    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:35.519677 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.519647    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:10:35.610459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.610425    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca\") pod \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") "
Nov 03 18:10:35.610693 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.610547    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l\" (UniqueName: \"kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l\") pod \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") "
Nov 03 18:10:35.610693 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.610673    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs\") pod \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") "
Nov 03 18:10:35.610861 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.610831    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\") pod \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\" (UID: \"3a79e47c-937a-4adb-b3db-fcb4c900aebf\") "
Nov 03 18:10:35.612324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.612297    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca" pod "3a79e47c-937a-4adb-b3db-fcb4c900aebf" (UID: "3a79e47c-937a-4adb-b3db-fcb4c900aebf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:10:35.614575 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.614541    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l" pod "3a79e47c-937a-4adb-b3db-fcb4c900aebf" (UID: "3a79e47c-937a-4adb-b3db-fcb4c900aebf"). InnerVolumeSpecName "kube-api-access-ktm4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:10:35.615112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.615087    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls" pod "3a79e47c-937a-4adb-b3db-fcb4c900aebf" (UID: "3a79e47c-937a-4adb-b3db-fcb4c900aebf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:35.615112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.615103    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs" pod "3a79e47c-937a-4adb-b3db-fcb4c900aebf" (UID: "3a79e47c-937a-4adb-b3db-fcb4c900aebf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:35.702429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.702389    5838 generic.go:356] "Generic (PLEG): container finished" podID="3a79e47c-937a-4adb-b3db-fcb4c900aebf" containerID="b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde" exitCode=0
Nov 03 18:10:35.702630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.702494    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"
Nov 03 18:10:35.702630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.702530    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" event={"ID":"3a79e47c-937a-4adb-b3db-fcb4c900aebf","Type":"ContainerDied","Data":"b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde"}
Nov 03 18:10:35.702630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.702585    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn" event={"ID":"3a79e47c-937a-4adb-b3db-fcb4c900aebf","Type":"ContainerDied","Data":"0617e51b1fddee62f86b73ab8cd89e28284474c5b7e6f9f551dc791cd3fe2003"}
Nov 03 18:10:35.702630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.702616    5838 scope.go:122] "RemoveContainer" containerID="b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde"
Nov 03 18:10:35.712431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.712390    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktm4l\" (UniqueName: \"kubernetes.io/projected/3a79e47c-937a-4adb-b3db-fcb4c900aebf-kube-api-access-ktm4l\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:35.712577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.712459    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:35.712577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.712473    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a79e47c-937a-4adb-b3db-fcb4c900aebf-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:35.712577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.712483    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a79e47c-937a-4adb-b3db-fcb4c900aebf-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:35.717046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.717027    5838 scope.go:122] "RemoveContainer" containerID="b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde"
Nov 03 18:10:35.717335 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:10:35.717312    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde\": container with ID starting with b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde not found: ID does not exist" containerID="b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde"
Nov 03 18:10:35.717411 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.717350    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde"} err="failed to get container status \"b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde\": rpc error: code = NotFound desc = could not find container \"b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde\": container with ID starting with b3ef63723a45c5a92dfa4884a0ec5069bdcc91a9fbcd73f884047796587eccde not found: ID does not exist"
Nov 03 18:10:35.884093 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.884016    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"]
Nov 03 18:10:35.887583 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:35.887557    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-btgrn"]
Nov 03 18:10:36.135074 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.134985    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a79e47c-937a-4adb-b3db-fcb4c900aebf" path="/var/lib/kubelet/pods/3a79e47c-937a-4adb-b3db-fcb4c900aebf/volumes"
Nov 03 18:10:36.403797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.403765    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"]
Nov 03 18:10:36.404229 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.403983    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" podUID="4036e36e-aa0a-4cf5-afda-77763c5a4dec" containerName="maas-api" containerID="cri-o://2ff935ba86b553eae8293029ef455c962a4ea1c64d48397018948dff34da164d" gracePeriod=30
Nov 03 18:10:36.504444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.504413    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:36.717139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.717104    5838 generic.go:356] "Generic (PLEG): container finished" podID="4036e36e-aa0a-4cf5-afda-77763c5a4dec" containerID="2ff935ba86b553eae8293029ef455c962a4ea1c64d48397018948dff34da164d" exitCode=0
Nov 03 18:10:36.717352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.717330    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" event={"ID":"4036e36e-aa0a-4cf5-afda-77763c5a4dec","Type":"ContainerDied","Data":"2ff935ba86b553eae8293029ef455c962a4ea1c64d48397018948dff34da164d"}
Nov 03 18:10:36.717431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.717390    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9" event={"ID":"4036e36e-aa0a-4cf5-afda-77763c5a4dec","Type":"ContainerDied","Data":"829e9319792f5502bd4867c7c79b007befa6435fd13a4fc7023ea43331b84fd2"}
Nov 03 18:10:36.717431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.717417    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829e9319792f5502bd4867c7c79b007befa6435fd13a4fc7023ea43331b84fd2"
Nov 03 18:10:36.733928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.733903    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:10:36.797601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.797555    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266" podUID="ab5f15eb-80fb-4d53-afe9-9d3891cbe655" containerName="payload-processing" containerID="cri-o://a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e" gracePeriod=25
Nov 03 18:10:36.848586 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.848553    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca\") pod \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") "
Nov 03 18:10:36.848868 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.848743    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6\" (UniqueName: \"kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6\") pod \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") "
Nov 03 18:10:36.848868 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.848777    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\") pod \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") "
Nov 03 18:10:36.848868 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.848841    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs\") pod \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\" (UID: \"4036e36e-aa0a-4cf5-afda-77763c5a4dec\") "
Nov 03 18:10:36.850514 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.850484    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca" pod "4036e36e-aa0a-4cf5-afda-77763c5a4dec" (UID: "4036e36e-aa0a-4cf5-afda-77763c5a4dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:10:36.852602 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.852571    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls" pod "4036e36e-aa0a-4cf5-afda-77763c5a4dec" (UID: "4036e36e-aa0a-4cf5-afda-77763c5a4dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:36.852719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.852652    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs" pod "4036e36e-aa0a-4cf5-afda-77763c5a4dec" (UID: "4036e36e-aa0a-4cf5-afda-77763c5a4dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:36.852888 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.852868    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6" pod "4036e36e-aa0a-4cf5-afda-77763c5a4dec" (UID: "4036e36e-aa0a-4cf5-afda-77763c5a4dec"). InnerVolumeSpecName "kube-api-access-545r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:10:36.887633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.886409    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:36.950701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.950608    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036e36e-aa0a-4cf5-afda-77763c5a4dec-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:36.950701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.950640    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-545r6\" (UniqueName: \"kubernetes.io/projected/4036e36e-aa0a-4cf5-afda-77763c5a4dec-kube-api-access-545r6\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:36.950701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.950654    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:36.950701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:36.950668    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4036e36e-aa0a-4cf5-afda-77763c5a4dec-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:37.080360 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.080331    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:10:37.154066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.154029    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf\" (UniqueName: \"kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf\") pod \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") "
Nov 03 18:10:37.154308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.154294    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume\") pod \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\" (UID: \"ab5f15eb-80fb-4d53-afe9-9d3891cbe655\") "
Nov 03 18:10:37.154937 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.154682    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume" pod "ab5f15eb-80fb-4d53-afe9-9d3891cbe655" (UID: "ab5f15eb-80fb-4d53-afe9-9d3891cbe655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:10:37.155271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.155247    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:37.156467 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.156445    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf" pod "ab5f15eb-80fb-4d53-afe9-9d3891cbe655" (UID: "ab5f15eb-80fb-4d53-afe9-9d3891cbe655"). InnerVolumeSpecName "kube-api-access-l9qmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:10:37.209615 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.209500    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:37.258191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.258148    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9qmf\" (UniqueName: \"kubernetes.io/projected/ab5f15eb-80fb-4d53-afe9-9d3891cbe655-kube-api-access-l9qmf\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:37.728601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.728564    5838 generic.go:356] "Generic (PLEG): container finished" podID="ab5f15eb-80fb-4d53-afe9-9d3891cbe655" containerID="a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e" exitCode=0
Nov 03 18:10:37.729117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.728614    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"
Nov 03 18:10:37.729117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.728654    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266" event={"ID":"ab5f15eb-80fb-4d53-afe9-9d3891cbe655","Type":"ContainerDied","Data":"a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e"}
Nov 03 18:10:37.729117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.728721    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266" event={"ID":"ab5f15eb-80fb-4d53-afe9-9d3891cbe655","Type":"ContainerDied","Data":"5b1800b47c93a29186d65a47f8cca6adaeb80e30ca899a0c96fc7e885862e117"}
Nov 03 18:10:37.729117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.728766    5838 scope.go:122] "RemoveContainer" containerID="a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e"
Nov 03 18:10:37.729117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.728786    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"
Nov 03 18:10:37.744036 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.744011    5838 scope.go:122] "RemoveContainer" containerID="a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e"
Nov 03 18:10:37.744355 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:10:37.744336    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e\": container with ID starting with a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e not found: ID does not exist" containerID="a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e"
Nov 03 18:10:37.744415 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.744365    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e"} err="failed to get container status \"a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e\": rpc error: code = NotFound desc = could not find container \"a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e\": container with ID starting with a4c0381b62062ffc1e9a9f7cdaff8eee1bfbba9ec61817fabda198e32f68924e not found: ID does not exist"
Nov 03 18:10:37.954007 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.953972    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"]
Nov 03 18:10:37.962850 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:37.962819    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17c6a-65ccf4bbb7-cr266"]
Nov 03 18:10:38.082986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.082901    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"]
Nov 03 18:10:38.089942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.089907    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c17c6a-6c4fdc6f6c-v92z9"]
Nov 03 18:10:38.118367 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.118335    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4036e36e-aa0a-4cf5-afda-77763c5a4dec" path="/var/lib/kubelet/pods/4036e36e-aa0a-4cf5-afda-77763c5a4dec/volumes"
Nov 03 18:10:38.118864 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.118849    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5f15eb-80fb-4d53-afe9-9d3891cbe655" path="/var/lib/kubelet/pods/ab5f15eb-80fb-4d53-afe9-9d3891cbe655/volumes"
Nov 03 18:10:38.230059 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.230015    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:38.438633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.438601    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:38.598428 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:38.598391    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:39.286823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:39.286788    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:40.352834 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:10:40.352795    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:10:43.202592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:43.202547    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:43.355170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:43.355138    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:43.718720 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:43.718692    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:43.993778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:43.993677    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:46.647940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:46.647904    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:46.803417 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:46.803382    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:47.059436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:47.059354    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:47.591370 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:47.591338    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:47.749784 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:47.749751    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:49.433625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:49.433588    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:50.011833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:50.011781    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:50.186831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:50.186796    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:54.868574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:54.868529    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:54.883422 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:54.883390    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"]
Nov 03 18:10:55.257278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:55.257239    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:55.414956 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:55.414920    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:55.644926 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:55.644891    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:59.516127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:59.516079    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:10:59.884155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:10:59.884064    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" podUID="97dd4294-32ab-48cc-904a-00dade8a2d63" containerName="payload-processing" containerID="cri-o://ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca" gracePeriod=25
Nov 03 18:11:00.090357 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:11:00.090324    5838 logging.go:55] [core] [Channel #3710 SubChannel #3711]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: connect: connection refused"
Nov 03 18:11:00.234246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.234212    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:11:00.343333 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.343300    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw\" (UniqueName: \"kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw\") pod \"97dd4294-32ab-48cc-904a-00dade8a2d63\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") "
Nov 03 18:11:00.343539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.343525    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume\") pod \"97dd4294-32ab-48cc-904a-00dade8a2d63\" (UID: \"97dd4294-32ab-48cc-904a-00dade8a2d63\") "
Nov 03 18:11:00.344163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.344101    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume" pod "97dd4294-32ab-48cc-904a-00dade8a2d63" (UID: "97dd4294-32ab-48cc-904a-00dade8a2d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:00.346131 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.346103    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw" pod "97dd4294-32ab-48cc-904a-00dade8a2d63" (UID: "97dd4294-32ab-48cc-904a-00dade8a2d63"). InnerVolumeSpecName "kube-api-access-n7mmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:00.445940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.445900    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mmw\" (UniqueName: \"kubernetes.io/projected/97dd4294-32ab-48cc-904a-00dade8a2d63-kube-api-access-n7mmw\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:00.445940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.445936    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dd4294-32ab-48cc-904a-00dade8a2d63-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:00.779557 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.779518    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:00.927916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.927879    5838 generic.go:356] "Generic (PLEG): container finished" podID="97dd4294-32ab-48cc-904a-00dade8a2d63" containerID="ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca" exitCode=0
Nov 03 18:11:00.928155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.927940    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"
Nov 03 18:11:00.928155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.927968    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" event={"ID":"97dd4294-32ab-48cc-904a-00dade8a2d63","Type":"ContainerDied","Data":"ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca"}
Nov 03 18:11:00.928155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.928030    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" event={"ID":"97dd4294-32ab-48cc-904a-00dade8a2d63","Type":"ContainerDied","Data":"6c1b1f19f3c4161bb4b0426ebe0419ab201443354cff8ac1fcc855963961ead5"}
Nov 03 18:11:00.928155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.928059    5838 scope.go:122] "RemoveContainer" containerID="ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca"
Nov 03 18:11:00.940445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.940423    5838 scope.go:122] "RemoveContainer" containerID="ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca"
Nov 03 18:11:00.940703 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:00.940682    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca\": container with ID starting with ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca not found: ID does not exist" containerID="ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca"
Nov 03 18:11:00.940824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:00.940716    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca"} err="failed to get container status \"ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca\": rpc error: code = NotFound desc = could not find container \"ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca\": container with ID starting with ad4df49c0c275a04346c0dfdc34d2016c6e16dd10260566b94bd2492e60afcca not found: ID does not exist"
Nov 03 18:11:01.090804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:01.090674    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l" podUID="97dd4294-32ab-48cc-904a-00dade8a2d63" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.117:50052\" within 1s: context deadline exceeded"
Nov 03 18:11:01.091050 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:11:01.090826    5838 logging.go:55] [core] [Channel #3710 SubChannel #3711]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: operation was canceled"
Nov 03 18:11:01.099009 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:01.098977    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"]
Nov 03 18:11:01.149341 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:01.149300    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8dfc64-586d5865cf-bgd5l"]
Nov 03 18:11:02.118949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:02.118915    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dd4294-32ab-48cc-904a-00dade8a2d63" path="/var/lib/kubelet/pods/97dd4294-32ab-48cc-904a-00dade8a2d63/volumes"
Nov 03 18:11:02.598613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:02.598575    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:02.757171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:02.757109    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:06.087940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:06.087862    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:10.340994 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:10.340950    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:11:10.857304 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:10.857267    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:12.237016 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:12.236964    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:12.591798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:12.591676    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:12.905637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:12.905579    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:13.588366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:13.588332    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:13.904544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:13.904505    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:14.219232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:14.219085    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:14.400072 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:14.400026    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:14.952106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:14.952068    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:19.834508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:19.834466    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:20.797419 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:20.797380    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:20.953152 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:20.953110    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:21.181351 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:21.181297    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:21.183780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:21.183757    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"]
Nov 03 18:11:21.183977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:21.183956    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" containerID="cri-o://ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95" gracePeriod=30
Nov 03 18:11:22.853753 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:22.853658    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:23.024043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:23.024007    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:23.067508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:23.067465    5838 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:11:23.067695 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:23.067535    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:11:23.176984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:23.176947    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"]
Nov 03 18:11:23.177201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:23.177172    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy" containerID="cri-o://0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0" gracePeriod=30
Nov 03 18:11:23.284755 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:23.284548    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:25.387500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:25.387422    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:26.459778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.459703    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:11:26.558844 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.558807    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.558855    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.558956    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.558992    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559028    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559068    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559136    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559185    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559266    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.559445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559294    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs\") pod \"0e177a1d-0f67-4cab-a5f9-19a269641e10\" (UID: \"0e177a1d-0f67-4cab-a5f9-19a269641e10\") "
Nov 03 18:11:26.559445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559298    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:26.559604 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559505    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.559604 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559542    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.559767 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.559705    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.560298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.560276    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.560355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.560305    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.560355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.560321    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.560355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.560336    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.560355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.560350    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e177a1d-0f67-4cab-a5f9-19a269641e10-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.561670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.561641    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "kube-api-access-9c79t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:26.561833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.561779    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.561972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.561950    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:11:26.562021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.561973    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token" pod "0e177a1d-0f67-4cab-a5f9-19a269641e10" (UID: "0e177a1d-0f67-4cab-a5f9-19a269641e10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:26.661543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.661503    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.661543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.661542    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.661804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.661558    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e177a1d-0f67-4cab-a5f9-19a269641e10-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.661804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:26.661573    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c79t\" (UniqueName: \"kubernetes.io/projected/0e177a1d-0f67-4cab-a5f9-19a269641e10-kube-api-access-9c79t\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:27.166878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.166837    5838 generic.go:356] "Generic (PLEG): container finished" podID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerID="ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95" exitCode=0
Nov 03 18:11:27.167105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.166885    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" event={"ID":"0e177a1d-0f67-4cab-a5f9-19a269641e10","Type":"ContainerDied","Data":"ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95"}
Nov 03 18:11:27.167105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.166920    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"
Nov 03 18:11:27.167105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.166966    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms" event={"ID":"0e177a1d-0f67-4cab-a5f9-19a269641e10","Type":"ContainerDied","Data":"ed6ae1fdcf0c0392b1344f2e62ca895a84adadc401e7e67715df149c844814a7"}
Nov 03 18:11:27.167105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.167000    5838 scope.go:122] "RemoveContainer" containerID="ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95"
Nov 03 18:11:27.184414 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.184370    5838 scope.go:122] "RemoveContainer" containerID="c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"
Nov 03 18:11:27.195495 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.195470    5838 scope.go:122] "RemoveContainer" containerID="ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95"
Nov 03 18:11:27.195853 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:27.195831    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95\": container with ID starting with ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95 not found: ID does not exist" containerID="ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95"
Nov 03 18:11:27.195926 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.195861    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95"} err="failed to get container status \"ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95\": rpc error: code = NotFound desc = could not find container \"ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95\": container with ID starting with ec7866172efd9ec86da4c10f88bcbcb91c87136655c4fd409e496aef21df7c95 not found: ID does not exist"
Nov 03 18:11:27.195926 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.195880    5838 scope.go:122] "RemoveContainer" containerID="c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"
Nov 03 18:11:27.196155 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:27.196141    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a\": container with ID starting with c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a not found: ID does not exist" containerID="c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"
Nov 03 18:11:27.196205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.196158    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a"} err="failed to get container status \"c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a\": rpc error: code = NotFound desc = could not find container \"c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a\": container with ID starting with c907b347d2be171ee9590c86953913f4702de73d1aa9c72ba86e8a6d1aa6f44a not found: ID does not exist"
Nov 03 18:11:27.341299 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.341260    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"]
Nov 03 18:11:27.388767 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:27.388719    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-v49ms"]
Nov 03 18:11:28.120536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.120493    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" path="/var/lib/kubelet/pods/0e177a1d-0f67-4cab-a5f9-19a269641e10/volumes"
Nov 03 18:11:28.479380 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.479352    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:11:28.586367 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586329    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.586367 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586374    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.586599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586415    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.586599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586508    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.586599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586582    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.586794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586709    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:28.586794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586719    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.586904 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586805    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:28.586948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586893    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.587023 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587003    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.587091 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.586879    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:28.587091 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587075    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:28.587205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587085    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd\") pod \"12acfe5b-5879-47ae-a456-0f50375d1b53\" (UID: \"12acfe5b-5879-47ae-a456-0f50375d1b53\") "
Nov 03 18:11:28.587533 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587362    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:28.587867 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587843    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.587961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587872    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.587961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587887    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12acfe5b-5879-47ae-a456-0f50375d1b53-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.587961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587902    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.587961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.587916    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.588960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.588938    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:11:28.589060 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.588982    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:28.589137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.589122    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:28.589438 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.589422    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd" pod "12acfe5b-5879-47ae-a456-0f50375d1b53" (UID: "12acfe5b-5879-47ae-a456-0f50375d1b53"). InnerVolumeSpecName "kube-api-access-v7lhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:28.689576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.689534    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7lhd\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-kube-api-access-v7lhd\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.689843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.689577    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.689843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.689603    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.689843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.689617    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12acfe5b-5879-47ae-a456-0f50375d1b53-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.790799 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.790699    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:28.948299 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:28.948268    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:29.187301 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.187260    5838 generic.go:356] "Generic (PLEG): container finished" podID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerID="0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0" exitCode=0
Nov 03 18:11:29.187863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.187318    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"
Nov 03 18:11:29.187863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.187341    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" event={"ID":"12acfe5b-5879-47ae-a456-0f50375d1b53","Type":"ContainerDied","Data":"0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0"}
Nov 03 18:11:29.187863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.187399    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66" event={"ID":"12acfe5b-5879-47ae-a456-0f50375d1b53","Type":"ContainerDied","Data":"0b0d0c3cbda0e51caa8e02fe9281705347b2dbf5d90100ea4fdc29e9c27fc42a"}
Nov 03 18:11:29.187863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.187427    5838 scope.go:122] "RemoveContainer" containerID="0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0"
Nov 03 18:11:29.199674 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.199654    5838 scope.go:122] "RemoveContainer" containerID="83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"
Nov 03 18:11:29.210634 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.210605    5838 scope.go:122] "RemoveContainer" containerID="0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0"
Nov 03 18:11:29.210953 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:29.210927    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0\": container with ID starting with 0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0 not found: ID does not exist" containerID="0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0"
Nov 03 18:11:29.211021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.210965    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0"} err="failed to get container status \"0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0\": rpc error: code = NotFound desc = could not find container \"0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0\": container with ID starting with 0aff3cc35287738ccf0e1cf16329f763b22a7b2cf9fc787f6d3cf7561f84a9c0 not found: ID does not exist"
Nov 03 18:11:29.211021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.210984    5838 scope.go:122] "RemoveContainer" containerID="83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"
Nov 03 18:11:29.211224 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:29.211206    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008\": container with ID starting with 83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008 not found: ID does not exist" containerID="83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"
Nov 03 18:11:29.211261 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.211232    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008"} err="failed to get container status \"83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008\": rpc error: code = NotFound desc = could not find container \"83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008\": container with ID starting with 83ee0499e07e4860868c4744d8331a6105ac6b474b72205d9ae2b5d4ed8ae008 not found: ID does not exist"
Nov 03 18:11:29.360319 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.360281    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"]
Nov 03 18:11:29.410661 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:29.410626    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d9a300-openshift-default-84489fdf6f-ljb66"]
Nov 03 18:11:30.122054 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:30.122000    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" path="/var/lib/kubelet/pods/12acfe5b-5879-47ae-a456-0f50375d1b53/volumes"
Nov 03 18:11:31.042648 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:31.042607    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:33.916494 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:33.916422    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:34.073258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:34.073222    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:35.847884 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:35.847847    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"]
Nov 03 18:11:35.848783 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:35.848708    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" podUID="cede57ba-951a-4f4b-a689-5cf9bfedc3dd" containerName="maas-api" containerID="cri-o://fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11" gracePeriod=30
Nov 03 18:11:35.953367 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:35.953331    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:36.180215 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.180185    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:11:36.210041 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.210010    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs\") pod \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") "
Nov 03 18:11:36.210233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.210084    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr\" (UniqueName: \"kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr\") pod \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") "
Nov 03 18:11:36.210233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.210177    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") pod \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") "
Nov 03 18:11:36.210352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.210249    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca\") pod \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\" (UID: \"cede57ba-951a-4f4b-a689-5cf9bfedc3dd\") "
Nov 03 18:11:36.210629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.210597    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca" pod "cede57ba-951a-4f4b-a689-5cf9bfedc3dd" (UID: "cede57ba-951a-4f4b-a689-5cf9bfedc3dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:36.211406 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.211380    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:36.212798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.212757    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr" pod "cede57ba-951a-4f4b-a689-5cf9bfedc3dd" (UID: "cede57ba-951a-4f4b-a689-5cf9bfedc3dd"). InnerVolumeSpecName "kube-api-access-bpcvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:36.212893 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.212873    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls" pod "cede57ba-951a-4f4b-a689-5cf9bfedc3dd" (UID: "cede57ba-951a-4f4b-a689-5cf9bfedc3dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:36.212962 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.212916    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs" pod "cede57ba-951a-4f4b-a689-5cf9bfedc3dd" (UID: "cede57ba-951a-4f4b-a689-5cf9bfedc3dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:36.280983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.280933    5838 generic.go:356] "Generic (PLEG): container finished" podID="cede57ba-951a-4f4b-a689-5cf9bfedc3dd" containerID="fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11" exitCode=0
Nov 03 18:11:36.280983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.280989    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"
Nov 03 18:11:36.281251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.280984    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" event={"ID":"cede57ba-951a-4f4b-a689-5cf9bfedc3dd","Type":"ContainerDied","Data":"fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11"}
Nov 03 18:11:36.282351 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.282315    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn" event={"ID":"cede57ba-951a-4f4b-a689-5cf9bfedc3dd","Type":"ContainerDied","Data":"033cc1fb622b5aa234b5aafa5bfa53e0155ef769bdd34055dcb1e61e9df995d7"}
Nov 03 18:11:36.282577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.282554    5838 scope.go:122] "RemoveContainer" containerID="fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11"
Nov 03 18:11:36.299438 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.299404    5838 scope.go:122] "RemoveContainer" containerID="fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11"
Nov 03 18:11:36.299917 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:36.299878    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11\": container with ID starting with fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11 not found: ID does not exist" containerID="fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11"
Nov 03 18:11:36.300071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.299930    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11"} err="failed to get container status \"fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11\": rpc error: code = NotFound desc = could not find container \"fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11\": container with ID starting with fc07fbf87a730d026244a8048328180512b14d982865cb6ba14ae3ee95844d11 not found: ID does not exist"
Nov 03 18:11:36.315781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.315697    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:36.315781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.315752    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcvr\" (UniqueName: \"kubernetes.io/projected/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-kube-api-access-bpcvr\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:36.315781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.315772    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede57ba-951a-4f4b-a689-5cf9bfedc3dd-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:36.428548 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.428446    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"]
Nov 03 18:11:36.433650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:36.433621    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e75517-69ff6f5d86-pbqxn"]
Nov 03 18:11:37.729198 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:37.729150    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:37.934304 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:37.934250    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:38.121848 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:38.121764    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cede57ba-951a-4f4b-a689-5cf9bfedc3dd" path="/var/lib/kubelet/pods/cede57ba-951a-4f4b-a689-5cf9bfedc3dd/volumes"
Nov 03 18:11:38.139142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:38.139069    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:38.823230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:38.823180    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:40.089168 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:40.089114    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:40.245133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:40.245091    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:40.348598 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:40.348505    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:11:43.488380 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:43.488351    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:43.643458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:43.643427    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:46.864753 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:46.864710    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:47.020777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:47.020721    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:47.267386 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:47.267356    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:48.761661 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:48.761633    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:48.919127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:48.919079    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:49.072054 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:49.071966    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:50.336883 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:50.336844    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:50.489122 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:50.489087    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:53.773656 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:53.773616    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"]
Nov 03 18:11:53.774166 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:53.773845    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4" podUID="9e64bcbf-4f4d-407e-981e-5d36b10ff380" containerName="maas-api" containerID="cri-o://406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e" gracePeriod=30
Nov 03 18:11:53.875688 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:53.875656    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:54.077311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.077283    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:11:54.094677 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.094636    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca\") pod \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") "
Nov 03 18:11:54.094861 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.094797    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j\" (UniqueName: \"kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j\") pod \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") "
Nov 03 18:11:54.094905 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.094859    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls\") pod \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") "
Nov 03 18:11:54.095044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.095026    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs\") pod \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\" (UID: \"9e64bcbf-4f4d-407e-981e-5d36b10ff380\") "
Nov 03 18:11:54.095101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.095069    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca" pod "9e64bcbf-4f4d-407e-981e-5d36b10ff380" (UID: "9e64bcbf-4f4d-407e-981e-5d36b10ff380"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:54.095537 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.095511    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e64bcbf-4f4d-407e-981e-5d36b10ff380-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:54.097246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.097201    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j" pod "9e64bcbf-4f4d-407e-981e-5d36b10ff380" (UID: "9e64bcbf-4f4d-407e-981e-5d36b10ff380"). InnerVolumeSpecName "kube-api-access-65c8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:54.097565 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.097543    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls" pod "9e64bcbf-4f4d-407e-981e-5d36b10ff380" (UID: "9e64bcbf-4f4d-407e-981e-5d36b10ff380"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:54.097696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.097669    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs" pod "9e64bcbf-4f4d-407e-981e-5d36b10ff380" (UID: "9e64bcbf-4f4d-407e-981e-5d36b10ff380"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:54.197249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.197195    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65c8j\" (UniqueName: \"kubernetes.io/projected/9e64bcbf-4f4d-407e-981e-5d36b10ff380-kube-api-access-65c8j\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:54.197249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.197242    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:54.197249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.197257    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e64bcbf-4f4d-407e-981e-5d36b10ff380-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:54.478261 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.478223    5838 generic.go:356] "Generic (PLEG): container finished" podID="9e64bcbf-4f4d-407e-981e-5d36b10ff380" containerID="406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e" exitCode=0
Nov 03 18:11:54.478455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.478280    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"
Nov 03 18:11:54.478455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.478305    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4" event={"ID":"9e64bcbf-4f4d-407e-981e-5d36b10ff380","Type":"ContainerDied","Data":"406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e"}
Nov 03 18:11:54.478455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.478363    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4" event={"ID":"9e64bcbf-4f4d-407e-981e-5d36b10ff380","Type":"ContainerDied","Data":"64798d4b0178ce2fdb3ddea6eb72a7126c2bd1990b79d718027a6bb1533d6789"}
Nov 03 18:11:54.478455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.478386    5838 scope.go:122] "RemoveContainer" containerID="406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e"
Nov 03 18:11:54.489223 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.489203    5838 scope.go:122] "RemoveContainer" containerID="406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e"
Nov 03 18:11:54.489542 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:11:54.489515    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e\": container with ID starting with 406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e not found: ID does not exist" containerID="406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e"
Nov 03 18:11:54.489658 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.489549    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e"} err="failed to get container status \"406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e\": rpc error: code = NotFound desc = could not find container \"406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e\": container with ID starting with 406ec62c6258ef3a094679ca5a55b452d3f4b0f5c0fe7ad21326205e1f81d26e not found: ID does not exist"
Nov 03 18:11:54.648364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.648324    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"]
Nov 03 18:11:54.656079 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:54.656051    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5100c-765f9796c6-ttvh4"]
Nov 03 18:11:55.776048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:55.776012    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:56.060435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:56.060340    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:56.132761 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:56.132708    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e64bcbf-4f4d-407e-981e-5d36b10ff380" path="/var/lib/kubelet/pods/9e64bcbf-4f4d-407e-981e-5d36b10ff380/volumes"
Nov 03 18:11:56.215813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:56.215774    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:56.780841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:56.780789    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:59.660112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:59.660080    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"]
Nov 03 18:11:59.770256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:11:59.770219    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"]
Nov 03 18:12:00.029665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:00.029577    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:00.183453 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:00.183420    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:01.889034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:01.888999    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:04.000298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.000208    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:04.199819 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.199711    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:04.357085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.356969    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:04.523177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.523147    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:04.660874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.660833    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps" podUID="23c89157-87af-4af2-a659-c6bac4bec72b" containerName="payload-processing" containerID="cri-o://fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363" gracePeriod=25
Nov 03 18:12:04.770694 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.770651    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc" podUID="c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" containerName="payload-pre-processing" containerID="cri-o://eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84" gracePeriod=25
Nov 03 18:12:04.954229 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.954201    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:12:04.969361 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.969326    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume\") pod \"23c89157-87af-4af2-a659-c6bac4bec72b\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") "
Nov 03 18:12:04.969539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.969395    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs\" (UniqueName: \"kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs\") pod \"23c89157-87af-4af2-a659-c6bac4bec72b\" (UID: \"23c89157-87af-4af2-a659-c6bac4bec72b\") "
Nov 03 18:12:04.970822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.970790    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume" pod "23c89157-87af-4af2-a659-c6bac4bec72b" (UID: "23c89157-87af-4af2-a659-c6bac4bec72b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:04.975596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:04.973903    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs" pod "23c89157-87af-4af2-a659-c6bac4bec72b" (UID: "23c89157-87af-4af2-a659-c6bac4bec72b"). InnerVolumeSpecName "kube-api-access-nj2zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:05.047992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.047963    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:12:05.071252 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.071221    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume\") pod \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") "
Nov 03 18:12:05.071428 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.071293    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh\" (UniqueName: \"kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh\") pod \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\" (UID: \"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99\") "
Nov 03 18:12:05.071607 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.071584    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume" pod "c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" (UID: "c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:05.071802 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.071787    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c89157-87af-4af2-a659-c6bac4bec72b-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:05.071847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.071805    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj2zs\" (UniqueName: \"kubernetes.io/projected/23c89157-87af-4af2-a659-c6bac4bec72b-kube-api-access-nj2zs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:05.071847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.071817    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:05.075172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.074551    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh" pod "c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" (UID: "c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99"). InnerVolumeSpecName "kube-api-access-vsnlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:05.173014 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.172918    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsnlh\" (UniqueName: \"kubernetes.io/projected/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99-kube-api-access-vsnlh\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:05.576268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.576171    5838 generic.go:356] "Generic (PLEG): container finished" podID="23c89157-87af-4af2-a659-c6bac4bec72b" containerID="fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363" exitCode=0
Nov 03 18:12:05.576268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.576245    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"
Nov 03 18:12:05.576549 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.576275    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps" event={"ID":"23c89157-87af-4af2-a659-c6bac4bec72b","Type":"ContainerDied","Data":"fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363"}
Nov 03 18:12:05.576549 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.576338    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps" event={"ID":"23c89157-87af-4af2-a659-c6bac4bec72b","Type":"ContainerDied","Data":"44813ff090f6e2a0e38c0c99e69e20b3d49d661e4dbee5489571fefc58eea6cf"}
Nov 03 18:12:05.576549 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.576372    5838 scope.go:122] "RemoveContainer" containerID="fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363"
Nov 03 18:12:05.579148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.579124    5838 generic.go:356] "Generic (PLEG): container finished" podID="c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" containerID="eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84" exitCode=0
Nov 03 18:12:05.579262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.579174    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"
Nov 03 18:12:05.579262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.579190    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc" event={"ID":"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99","Type":"ContainerDied","Data":"eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84"}
Nov 03 18:12:05.579354 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.579263    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc" event={"ID":"c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99","Type":"ContainerDied","Data":"9ea0d667285d74fd91212de648f64b33f76c2d258ea96db2934489724e8b2d63"}
Nov 03 18:12:05.595025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.592489    5838 scope.go:122] "RemoveContainer" containerID="fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363"
Nov 03 18:12:05.595025 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:05.593094    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363\": container with ID starting with fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363 not found: ID does not exist" containerID="fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363"
Nov 03 18:12:05.595025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.593147    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363"} err="failed to get container status \"fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363\": rpc error: code = NotFound desc = could not find container \"fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363\": container with ID starting with fcd112732addbc83966cbff8ea3977929cfa539b6eb5c7467ba12bc7f0bc2363 not found: ID does not exist"
Nov 03 18:12:05.595025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.593172    5838 scope.go:122] "RemoveContainer" containerID="eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84"
Nov 03 18:12:05.612429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.612386    5838 scope.go:122] "RemoveContainer" containerID="eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84"
Nov 03 18:12:05.612847 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:05.612816    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84\": container with ID starting with eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84 not found: ID does not exist" containerID="eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84"
Nov 03 18:12:05.612948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.612862    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84"} err="failed to get container status \"eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84\": rpc error: code = NotFound desc = could not find container \"eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84\": container with ID starting with eb39e02994f63fc6d2792951edc8075e506808e20e1bbf7d1ac18ffe424fbd84 not found: ID does not exist"
Nov 03 18:12:05.749587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.749549    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"]
Nov 03 18:12:05.797322 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.797280    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62aead-588d95f44c-8vwnc"]
Nov 03 18:12:05.870151 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.870057    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"]
Nov 03 18:12:05.878095 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:05.878065    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62aead-765f48d57c-shbps"]
Nov 03 18:12:06.117292 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.117260    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c89157-87af-4af2-a659-c6bac4bec72b" path="/var/lib/kubelet/pods/23c89157-87af-4af2-a659-c6bac4bec72b/volumes"
Nov 03 18:12:06.117719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.117704    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" path="/var/lib/kubelet/pods/c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99/volumes"
Nov 03 18:12:06.603726 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.603688    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:06.943798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.943765    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"]
Nov 03 18:12:06.944618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944596    5838 cpu_manager.go:414] "containerMap: removing container" podUID="97dd4294-32ab-48cc-904a-00dade8a2d63" containerName="payload-processing"
Nov 03 18:12:06.944618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944618    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dd4294-32ab-48cc-904a-00dade8a2d63" containerName="payload-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944637    5838 cpu_manager.go:414] "containerMap: removing container" podUID="23c89157-87af-4af2-a659-c6bac4bec72b" containerName="payload-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944643    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c89157-87af-4af2-a659-c6bac4bec72b" containerName="payload-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944651    5838 cpu_manager.go:414] "containerMap: removing container" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944657    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944669    5838 cpu_manager.go:414] "containerMap: removing container" podUID="9e64bcbf-4f4d-407e-981e-5d36b10ff380" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944674    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e64bcbf-4f4d-407e-981e-5d36b10ff380" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944685    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3a79e47c-937a-4adb-b3db-fcb4c900aebf" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944690    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a79e47c-937a-4adb-b3db-fcb4c900aebf" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944700    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ab5f15eb-80fb-4d53-afe9-9d3891cbe655" containerName="payload-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944704    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5f15eb-80fb-4d53-afe9-9d3891cbe655" containerName="payload-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944715    5838 cpu_manager.go:414] "containerMap: removing container" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944719    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944748    5838 cpu_manager.go:414] "containerMap: removing container" podUID="cede57ba-951a-4f4b-a689-5cf9bfedc3dd" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944753    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cede57ba-951a-4f4b-a689-5cf9bfedc3dd" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944770    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" containerName="payload-pre-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944775    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" containerName="payload-pre-processing"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944783    5838 cpu_manager.go:414] "containerMap: removing container" podUID="4036e36e-aa0a-4cf5-afda-77763c5a4dec" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944787    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4036e36e-aa0a-4cf5-afda-77763c5a4dec" containerName="maas-api"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944795    5838 cpu_manager.go:414] "containerMap: removing container" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944800    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944811    5838 cpu_manager.go:414] "containerMap: removing container" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy"
Nov 03 18:12:06.944813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944817    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944924    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e9aa53-7fd0-4178-8c5c-e0d65fd66f99" containerName="payload-pre-processing"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944936    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="4036e36e-aa0a-4cf5-afda-77763c5a4dec" containerName="maas-api"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944943    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944951    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e177a1d-0f67-4cab-a5f9-19a269641e10" containerName="istio-proxy"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944958    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dd4294-32ab-48cc-904a-00dade8a2d63" containerName="payload-processing"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944965    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c89157-87af-4af2-a659-c6bac4bec72b" containerName="payload-processing"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944971    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944983    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e64bcbf-4f4d-407e-981e-5d36b10ff380" containerName="maas-api"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944991    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a79e47c-937a-4adb-b3db-fcb4c900aebf" containerName="maas-api"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.944998    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5f15eb-80fb-4d53-afe9-9d3891cbe655" containerName="payload-processing"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.945005    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="12acfe5b-5879-47ae-a456-0f50375d1b53" containerName="istio-proxy"
Nov 03 18:12:06.945484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.945011    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="cede57ba-951a-4f4b-a689-5cf9bfedc3dd" containerName="maas-api"
Nov 03 18:12:06.952181 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:06.952157    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.006439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.006402    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vfc\" (UniqueName: \"kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.026512 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.026474    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.026724 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.026536    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.026724 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.026571    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.072356 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.072320    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6ed3e95c-serving-cert\""
Nov 03 18:12:07.075977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.075948    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"]
Nov 03 18:12:07.128219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.128168    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vfc\" (UniqueName: \"kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.128701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.128319    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.128701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.128374    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.128701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.128419    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.129009 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:07.128784    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-serving-cert: secret "maas-api-e2e-mt-6ed3e95c-serving-cert" not found
Nov 03 18:12:07.129009 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:07.128869    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls podName:68411659-153d-4907-90d4-4e34648af4f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:12:07.628846235 +0000 UTC m=+2263.639584316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls") pod "maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" (UID: "68411659-153d-4907-90d4-4e34648af4f5") : secret "maas-api-e2e-mt-6ed3e95c-serving-cert" not found
Nov 03 18:12:07.129481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.129452    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.131379 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.131355    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.171284 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.171249    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:07.192332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.192289    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vfc\" (UniqueName: \"kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.337856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.337721    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:07.493439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.493395    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:07.641665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.641627    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.644279 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.644255    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\") pod \"maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:07.650547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.650495    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:07.802423 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.802392    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:07.866547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:07.866510    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:08.254484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:08.254452    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"]
Nov 03 18:12:08.255326 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:12:08.255301    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68411659_153d_4907_90d4_4e34648af4f5.slice/crio-25a8585cd28cae1d97d924e67796c5bb68b37be97dd2b89d146dc01df691d290 WatchSource:0}: Error finding container 25a8585cd28cae1d97d924e67796c5bb68b37be97dd2b89d146dc01df691d290: Status 404 returned error can't find the container with id 25a8585cd28cae1d97d924e67796c5bb68b37be97dd2b89d146dc01df691d290
Nov 03 18:12:08.608018 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:08.607930    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" event={"ID":"68411659-153d-4907-90d4-4e34648af4f5","Type":"ContainerStarted","Data":"25a8585cd28cae1d97d924e67796c5bb68b37be97dd2b89d146dc01df691d290"}
Nov 03 18:12:09.619230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:09.619178    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" event={"ID":"68411659-153d-4907-90d4-4e34648af4f5","Type":"ContainerStarted","Data":"8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a"}
Nov 03 18:12:09.619687 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:09.619286    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:09.740603 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:09.740545    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" podStartSLOduration=2.928849446 podStartE2EDuration="3.740530829s" totalImagesPullingTime="811.681383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:09.74004527 +0000 UTC m=+2265.750783362" watchObservedRunningTime="2026-11-03 18:12:09.740530829 +0000 UTC m=+2265.751268901"
Nov 03 18:12:10.343319 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:10.343276    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:12:11.864352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:11.864315    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:12.039294 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:12.039249    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:12.208917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:12.208883    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:12.729939 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:12.729907    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:15.633325 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:15.633290    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:12:15.984419 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:15.984372    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:16.160832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:16.160713    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:16.291622 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:16.291520    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"]
Nov 03 18:12:16.291873 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:16.291726    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy" containerID="cri-o://80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd" gracePeriod=30
Nov 03 18:12:17.575229 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:17.575185    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:19.304990 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:19.304948    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:19.532539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:19.532489    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:19.686634 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:19.686585    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:20.345559 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:20.345527    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:21.574476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.574411    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 18:12:21.642050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642018    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642092    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642127    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642217    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642421 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642256    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642421 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642337    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642421 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642336    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:21.642421 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642398    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:21.642635 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642476    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642635 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642591    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642635 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642630    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket\") pod \"a78778b6-1409-4437-b7eb-8de9aea31362\" (UID: \"a78778b6-1409-4437-b7eb-8de9aea31362\") "
Nov 03 18:12:21.642846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642664    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:21.642989 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.642949    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:21.643424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.643313    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78778b6-1409-4437-b7eb-8de9aea31362-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.643424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.643340    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.643424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.643339    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:21.643424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.643354    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.643424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.643368    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.645097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.645066    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:21.645196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.645165    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:21.645196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.645171    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:21.645938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.645913    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw" pod "a78778b6-1409-4437-b7eb-8de9aea31362" (UID: "a78778b6-1409-4437-b7eb-8de9aea31362"). InnerVolumeSpecName "kube-api-access-pjpfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:21.733650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.733613    5838 generic.go:356] "Generic (PLEG): container finished" podID="a78778b6-1409-4437-b7eb-8de9aea31362" containerID="80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd" exitCode=0
Nov 03 18:12:21.733852 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.733665    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" event={"ID":"a78778b6-1409-4437-b7eb-8de9aea31362","Type":"ContainerDied","Data":"80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd"}
Nov 03 18:12:21.733852 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.733706    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"
Nov 03 18:12:21.733852 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.733721    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w" event={"ID":"a78778b6-1409-4437-b7eb-8de9aea31362","Type":"ContainerDied","Data":"f1f37b590cb6dcecd64af3886207a8854744c346ba53cbcd1f17a5444fc01930"}
Nov 03 18:12:21.733852 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.733776    5838 scope.go:122] "RemoveContainer" containerID="80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd"
Nov 03 18:12:21.745067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.745037    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.745067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.745064    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78778b6-1409-4437-b7eb-8de9aea31362-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.745290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.745077    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjpfw\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-kube-api-access-pjpfw\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.745290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.745087    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78778b6-1409-4437-b7eb-8de9aea31362-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.745290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.745097    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78778b6-1409-4437-b7eb-8de9aea31362-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:21.747417 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.747391    5838 scope.go:122] "RemoveContainer" containerID="80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd"
Nov 03 18:12:21.748016 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:21.747994    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd\": container with ID starting with 80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd not found: ID does not exist" containerID="80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd"
Nov 03 18:12:21.748121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.748022    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd"} err="failed to get container status \"80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd\": rpc error: code = NotFound desc = could not find container \"80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd\": container with ID starting with 80a9bbd56945bbf929aaa01f180f39a1a610e524249c7a8f8dd7bb5de18b1fbd not found: ID does not exist"
Nov 03 18:12:21.865588 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.865494    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"]
Nov 03 18:12:21.871098 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:21.871071    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-h8w4w"]
Nov 03 18:12:22.121235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:22.121146    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" path="/var/lib/kubelet/pods/a78778b6-1409-4437-b7eb-8de9aea31362/volumes"
Nov 03 18:12:24.911236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:24.911198    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:26.134037 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:26.134003    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:12:27.814760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:27.814705    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:27.969485 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:27.969447    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:28.486541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:28.486469    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:28.640329 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:28.640290    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:28.892340 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:28.892304    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:30.112587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:30.112556    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:35.394428 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:35.394392    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:40.019160 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.019120    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"]
Nov 03 18:12:40.019842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.019337    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" podUID="3baa4862-c70f-405b-a50f-b3c1664e249d" containerName="maas-api" containerID="cri-o://9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d" gracePeriod=30
Nov 03 18:12:40.120223 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.120188    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:40.339279 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.339245    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:12:40.343926 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:40.343892    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:12:40.395792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.395292    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs\") pod \"3baa4862-c70f-405b-a50f-b3c1664e249d\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") "
Nov 03 18:12:40.395792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.395371    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\") pod \"3baa4862-c70f-405b-a50f-b3c1664e249d\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") "
Nov 03 18:12:40.395792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.395403    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca\") pod \"3baa4862-c70f-405b-a50f-b3c1664e249d\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") "
Nov 03 18:12:40.395792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.395489    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th\" (UniqueName: \"kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th\") pod \"3baa4862-c70f-405b-a50f-b3c1664e249d\" (UID: \"3baa4862-c70f-405b-a50f-b3c1664e249d\") "
Nov 03 18:12:40.416550 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.396999    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca" pod "3baa4862-c70f-405b-a50f-b3c1664e249d" (UID: "3baa4862-c70f-405b-a50f-b3c1664e249d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:40.417455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.417414    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs" pod "3baa4862-c70f-405b-a50f-b3c1664e249d" (UID: "3baa4862-c70f-405b-a50f-b3c1664e249d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:40.418910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.418880    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls" pod "3baa4862-c70f-405b-a50f-b3c1664e249d" (UID: "3baa4862-c70f-405b-a50f-b3c1664e249d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:40.419048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.419029    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th" pod "3baa4862-c70f-405b-a50f-b3c1664e249d" (UID: "3baa4862-c70f-405b-a50f-b3c1664e249d"). InnerVolumeSpecName "kube-api-access-hw7th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:40.497453 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.497420    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:40.497453 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.497451    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baa4862-c70f-405b-a50f-b3c1664e249d-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:40.497718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.497461    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7th\" (UniqueName: \"kubernetes.io/projected/3baa4862-c70f-405b-a50f-b3c1664e249d-kube-api-access-hw7th\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:40.497718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.497471    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3baa4862-c70f-405b-a50f-b3c1664e249d-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:40.926894 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.926847    5838 generic.go:356] "Generic (PLEG): container finished" podID="3baa4862-c70f-405b-a50f-b3c1664e249d" containerID="9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d" exitCode=0
Nov 03 18:12:40.927113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.926927    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"
Nov 03 18:12:40.927113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.926947    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" event={"ID":"3baa4862-c70f-405b-a50f-b3c1664e249d","Type":"ContainerDied","Data":"9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d"}
Nov 03 18:12:40.927113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.927046    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs" event={"ID":"3baa4862-c70f-405b-a50f-b3c1664e249d","Type":"ContainerDied","Data":"6dd3f19fe0f51f28c3facd5a4f5e73a188e336df800e6bf7bb698b53082d9b0f"}
Nov 03 18:12:40.927113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.927078    5838 scope.go:122] "RemoveContainer" containerID="9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d"
Nov 03 18:12:40.939907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.939883    5838 scope.go:122] "RemoveContainer" containerID="9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d"
Nov 03 18:12:40.940258 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:40.940236    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d\": container with ID starting with 9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d not found: ID does not exist" containerID="9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d"
Nov 03 18:12:40.940331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:40.940269    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d"} err="failed to get container status \"9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d\": rpc error: code = NotFound desc = could not find container \"9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d\": container with ID starting with 9906ef55ee54bdc65747f35d8c9991d666edf68e7a79901ccfb2d1815e9d795d not found: ID does not exist"
Nov 03 18:12:41.100587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.100553    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"]
Nov 03 18:12:41.103826 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.103801    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2569c8-7d9dbc4f7b-sb9qs"]
Nov 03 18:12:41.324723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.324627    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"]
Nov 03 18:12:41.324942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.324919    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" podUID="b1bd643f-bde2-4d20-adc5-7e22da9306e9" containerName="maas-api" containerID="cri-o://2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4" gracePeriod=30
Nov 03 18:12:41.430030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.429997    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:41.440964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.440934    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"]
Nov 03 18:12:41.653146 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.653106    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:12:41.699808 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.699769    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:41.721950 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.721913    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca\") pod \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") "
Nov 03 18:12:41.722170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.722099    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\") pod \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") "
Nov 03 18:12:41.722267 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.722242    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp\" (UniqueName: \"kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp\") pod \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") "
Nov 03 18:12:41.722335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.722276    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs\") pod \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\" (UID: \"b1bd643f-bde2-4d20-adc5-7e22da9306e9\") "
Nov 03 18:12:41.722397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.722365    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca" pod "b1bd643f-bde2-4d20-adc5-7e22da9306e9" (UID: "b1bd643f-bde2-4d20-adc5-7e22da9306e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:41.722896 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.722818    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1bd643f-bde2-4d20-adc5-7e22da9306e9-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:41.724470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.724437    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls" pod "b1bd643f-bde2-4d20-adc5-7e22da9306e9" (UID: "b1bd643f-bde2-4d20-adc5-7e22da9306e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:41.724577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.724480    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp" pod "b1bd643f-bde2-4d20-adc5-7e22da9306e9" (UID: "b1bd643f-bde2-4d20-adc5-7e22da9306e9"). InnerVolumeSpecName "kube-api-access-cp9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:41.724617 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.724580    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs" pod "b1bd643f-bde2-4d20-adc5-7e22da9306e9" (UID: "b1bd643f-bde2-4d20-adc5-7e22da9306e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:41.825851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.825010    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9hp\" (UniqueName: \"kubernetes.io/projected/b1bd643f-bde2-4d20-adc5-7e22da9306e9-kube-api-access-cp9hp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:41.825851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.825053    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:41.825851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.825069    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1bd643f-bde2-4d20-adc5-7e22da9306e9-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:41.856440 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.856330    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:41.941172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.941133    5838 generic.go:356] "Generic (PLEG): container finished" podID="b1bd643f-bde2-4d20-adc5-7e22da9306e9" containerID="2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4" exitCode=0
Nov 03 18:12:41.941368 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.941235    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" event={"ID":"b1bd643f-bde2-4d20-adc5-7e22da9306e9","Type":"ContainerDied","Data":"2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4"}
Nov 03 18:12:41.941368 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.941289    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr" event={"ID":"b1bd643f-bde2-4d20-adc5-7e22da9306e9","Type":"ContainerDied","Data":"6f7598492ab29beac32dad420cc4eabb7ed397b4039bfa93bd677ab02e58e561"}
Nov 03 18:12:41.941368 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.941320    5838 scope.go:122] "RemoveContainer" containerID="2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4"
Nov 03 18:12:41.941508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.941433    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"
Nov 03 18:12:41.967659 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.967630    5838 scope.go:122] "RemoveContainer" containerID="2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4"
Nov 03 18:12:41.968081 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:41.968060    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4\": container with ID starting with 2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4 not found: ID does not exist" containerID="2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4"
Nov 03 18:12:41.968166 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:41.968090    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4"} err="failed to get container status \"2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4\": rpc error: code = NotFound desc = could not find container \"2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4\": container with ID starting with 2ce18eec0e2bf17cae6d5fc6d42b2f24ab0333c630b040570e5fc5ca7093e2b4 not found: ID does not exist"
Nov 03 18:12:42.118823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:42.118722    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3baa4862-c70f-405b-a50f-b3c1664e249d" path="/var/lib/kubelet/pods/3baa4862-c70f-405b-a50f-b3c1664e249d/volumes"
Nov 03 18:12:42.119221 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:42.119073    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"]
Nov 03 18:12:42.119221 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:42.119101    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-19d275-c6b796d6c-hcxgr"]
Nov 03 18:12:42.128391 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:42.128361    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:42.825310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:42.825237    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:43.132688 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:43.132563    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:43.579695 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:43.579640    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:43.731618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:43.731585    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:43.886961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:43.886838    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:44.121466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:44.121429    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1bd643f-bde2-4d20-adc5-7e22da9306e9" path="/var/lib/kubelet/pods/b1bd643f-bde2-4d20-adc5-7e22da9306e9/volumes"
Nov 03 18:12:44.251124 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:44.251096    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:46.442082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.442035    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx" podUID="5400cffc-6eb1-4f0a-9a60-eef0f0e27666" containerName="payload-pre-processing" containerID="cri-o://ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c" gracePeriod=25
Nov 03 18:12:46.728376 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.728346    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:12:46.813121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.813065    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn\" (UniqueName: \"kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn\") pod \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") "
Nov 03 18:12:46.813344 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.813281    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume\") pod \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\" (UID: \"5400cffc-6eb1-4f0a-9a60-eef0f0e27666\") "
Nov 03 18:12:46.813697 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.813672    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume" pod "5400cffc-6eb1-4f0a-9a60-eef0f0e27666" (UID: "5400cffc-6eb1-4f0a-9a60-eef0f0e27666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:46.813834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.813814    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:46.815432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.815407    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn" pod "5400cffc-6eb1-4f0a-9a60-eef0f0e27666" (UID: "5400cffc-6eb1-4f0a-9a60-eef0f0e27666"). InnerVolumeSpecName "kube-api-access-55mdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:46.915193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.915142    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55mdn\" (UniqueName: \"kubernetes.io/projected/5400cffc-6eb1-4f0a-9a60-eef0f0e27666-kube-api-access-55mdn\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:46.980231 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.980149    5838 generic.go:356] "Generic (PLEG): container finished" podID="5400cffc-6eb1-4f0a-9a60-eef0f0e27666" containerID="ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c" exitCode=0
Nov 03 18:12:46.980231 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.980206    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"
Nov 03 18:12:46.980437 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.980229    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx" event={"ID":"5400cffc-6eb1-4f0a-9a60-eef0f0e27666","Type":"ContainerDied","Data":"ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c"}
Nov 03 18:12:46.980437 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.980291    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx" event={"ID":"5400cffc-6eb1-4f0a-9a60-eef0f0e27666","Type":"ContainerDied","Data":"d799b7c246af6fa0d605614392a2adee9beba54f184d0d95ac11fcc1f9278904"}
Nov 03 18:12:46.980437 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.980321    5838 scope.go:122] "RemoveContainer" containerID="ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c"
Nov 03 18:12:46.992138 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.992113    5838 scope.go:122] "RemoveContainer" containerID="ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c"
Nov 03 18:12:46.992447 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:12:46.992421    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c\": container with ID starting with ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c not found: ID does not exist" containerID="ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c"
Nov 03 18:12:46.992517 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:46.992455    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c"} err="failed to get container status \"ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c\": rpc error: code = NotFound desc = could not find container \"ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c\": container with ID starting with ad7d75b7b4ac765d914e9d494ae5c7be416f4909c88329fb28f548f32304cf7c not found: ID does not exist"
Nov 03 18:12:47.150366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:47.150323    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"]
Nov 03 18:12:47.157693 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:47.157655    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-528e53-b5586fc9f-hgvjx"]
Nov 03 18:12:48.118994 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:48.118961    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5400cffc-6eb1-4f0a-9a60-eef0f0e27666" path="/var/lib/kubelet/pods/5400cffc-6eb1-4f0a-9a60-eef0f0e27666/volumes"
Nov 03 18:12:48.990797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:48.990760    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:49.194725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.194691    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"]
Nov 03 18:12:49.195328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195314    5838 cpu_manager.go:414] "containerMap: removing container" podUID="5400cffc-6eb1-4f0a-9a60-eef0f0e27666" containerName="payload-pre-processing"
Nov 03 18:12:49.195375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195330    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5400cffc-6eb1-4f0a-9a60-eef0f0e27666" containerName="payload-pre-processing"
Nov 03 18:12:49.195375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195354    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy"
Nov 03 18:12:49.195375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195359    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy"
Nov 03 18:12:49.195375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195373    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3baa4862-c70f-405b-a50f-b3c1664e249d" containerName="maas-api"
Nov 03 18:12:49.195505 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195378    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baa4862-c70f-405b-a50f-b3c1664e249d" containerName="maas-api"
Nov 03 18:12:49.195505 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195394    5838 cpu_manager.go:414] "containerMap: removing container" podUID="b1bd643f-bde2-4d20-adc5-7e22da9306e9" containerName="maas-api"
Nov 03 18:12:49.195505 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195399    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1bd643f-bde2-4d20-adc5-7e22da9306e9" containerName="maas-api"
Nov 03 18:12:49.195606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195508    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baa4862-c70f-405b-a50f-b3c1664e249d" containerName="maas-api"
Nov 03 18:12:49.195606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195518    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1bd643f-bde2-4d20-adc5-7e22da9306e9" containerName="maas-api"
Nov 03 18:12:49.195606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195531    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5400cffc-6eb1-4f0a-9a60-eef0f0e27666" containerName="payload-pre-processing"
Nov 03 18:12:49.195606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.195540    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78778b6-1409-4437-b7eb-8de9aea31362" containerName="istio-proxy"
Nov 03 18:12:49.200303 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.200279    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.254065 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.253977    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6ed3e95c-openshift-default-dockercfg-rk4jm\""
Nov 03 18:12:49.350315 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350272    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350315 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350319    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350439    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350477    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350507    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350667 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350555    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350652    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350702    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.350836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.350787    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkbg\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.415853 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.415814    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"]
Nov 03 18:12:49.451923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.451885    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.451937    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.451974    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452000    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452039    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452070    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452095    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkbg\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452296    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452335    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452606    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452754    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452886    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.452999 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.452920    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.453453 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.453426    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.455268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.455247    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.455383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.455312    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.516086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.516003    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.516235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.516106    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkbg\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:49.814647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:49.814555    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:50.262777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:50.262746    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"]
Nov 03 18:12:50.264180 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:12:50.264157    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b48127_ba3d_4760_9c93_a497c27e9321.slice/crio-83d4596f890953a9e130acbeadd44653ed0ebe2cb1e50a73a44683f282af2123 WatchSource:0}: Error finding container 83d4596f890953a9e130acbeadd44653ed0ebe2cb1e50a73a44683f282af2123: Status 404 returned error can't find the container with id 83d4596f890953a9e130acbeadd44653ed0ebe2cb1e50a73a44683f282af2123
Nov 03 18:12:50.266724 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:50.266685    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:12:50.266833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:50.266777    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:12:50.266833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:50.266810    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:12:50.922895 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:50.922842    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:51.033064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.033022    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" event={"ID":"e3b48127-ba3d-4760-9c93-a497c27e9321","Type":"ContainerStarted","Data":"8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4"}
Nov 03 18:12:51.033369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.033330    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" event={"ID":"e3b48127-ba3d-4760-9c93-a497c27e9321","Type":"ContainerStarted","Data":"83d4596f890953a9e130acbeadd44653ed0ebe2cb1e50a73a44683f282af2123"}
Nov 03 18:12:51.080045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.080011    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:51.217431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.211822    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podStartSLOduration=2.211801774 podStartE2EDuration="2.211801774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:51.211264539 +0000 UTC m=+2307.222002629" watchObservedRunningTime="2026-11-03 18:12:51.211801774 +0000 UTC m=+2307.222539847"
Nov 03 18:12:51.324439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.324379    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:51.818547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.818511    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:51.818991 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.818963    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 18:12:51.819169 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:51.819138    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 18:12:52.216672 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.216514    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:52.370066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.370019    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:52.527628 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.527533    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:52.752839 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.752793    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:52.828783 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.828145    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 18:12:52.828783 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.828210    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 18:12:52.984631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:52.983657    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:53.474997 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:53.474953    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:53.813819 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:53.810298    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:53.824760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:53.822267    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 18:12:53.824760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:53.822353    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 18:12:53.964395 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:53.964356    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:54.307760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:54.305297    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:54.820760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:54.817842    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 18:12:54.820760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:54.817909    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 18:12:55.816140 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:55.816099    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 18:12:55.816479 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:55.816444    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 18:12:56.820975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:56.820924    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:56.821455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:56.821326    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:56.822132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:56.822114    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:12:57.441253 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:57.441221    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:57.621070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:57.621036    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:57.780706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:57.780615    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:58.841409 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:58.841362    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"]
Nov 03 18:12:58.841886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:58.841568    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" podUID="041f2c85-51cd-40e8-9931-625774cbe220" containerName="maas-api" containerID="cri-o://58a482dc2a7b39bc21d85ec8d83841d6078e21dc8f1275606a9dec50c75dd354" gracePeriod=30
Nov 03 18:12:58.943939 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:58.943885    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:59.137533 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.137418    5838 generic.go:356] "Generic (PLEG): container finished" podID="041f2c85-51cd-40e8-9931-625774cbe220" containerID="58a482dc2a7b39bc21d85ec8d83841d6078e21dc8f1275606a9dec50c75dd354" exitCode=0
Nov 03 18:12:59.137759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.137587    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" event={"ID":"041f2c85-51cd-40e8-9931-625774cbe220","Type":"ContainerDied","Data":"58a482dc2a7b39bc21d85ec8d83841d6078e21dc8f1275606a9dec50c75dd354"}
Nov 03 18:12:59.193213 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.193187    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:12:59.341717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.340202    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs\") pod \"041f2c85-51cd-40e8-9931-625774cbe220\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") "
Nov 03 18:12:59.343441 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.343389    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2\" (UniqueName: \"kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2\") pod \"041f2c85-51cd-40e8-9931-625774cbe220\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") "
Nov 03 18:12:59.343687 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.343669    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\") pod \"041f2c85-51cd-40e8-9931-625774cbe220\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") "
Nov 03 18:12:59.344036 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.344014    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca\") pod \"041f2c85-51cd-40e8-9931-625774cbe220\" (UID: \"041f2c85-51cd-40e8-9931-625774cbe220\") "
Nov 03 18:12:59.346650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.346602    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca" pod "041f2c85-51cd-40e8-9931-625774cbe220" (UID: "041f2c85-51cd-40e8-9931-625774cbe220"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:59.374876 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.374833    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs" pod "041f2c85-51cd-40e8-9931-625774cbe220" (UID: "041f2c85-51cd-40e8-9931-625774cbe220"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:59.377270 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.377219    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2" pod "041f2c85-51cd-40e8-9931-625774cbe220" (UID: "041f2c85-51cd-40e8-9931-625774cbe220"). InnerVolumeSpecName "kube-api-access-6l7c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:59.377939 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.377912    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls" pod "041f2c85-51cd-40e8-9931-625774cbe220" (UID: "041f2c85-51cd-40e8-9931-625774cbe220"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:59.450313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.450271    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/041f2c85-51cd-40e8-9931-625774cbe220-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:59.450313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.450312    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:59.450614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.450329    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l7c2\" (UniqueName: \"kubernetes.io/projected/041f2c85-51cd-40e8-9931-625774cbe220-kube-api-access-6l7c2\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:59.450614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:12:59.450343    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/041f2c85-51cd-40e8-9931-625774cbe220-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:00.147461 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.147410    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc" event={"ID":"041f2c85-51cd-40e8-9931-625774cbe220","Type":"ContainerDied","Data":"2ae02a72b6e2850614b8e10becc5b18741d7abda1724bf942af7fad20d302f0d"}
Nov 03 18:13:00.147915 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.147493    5838 scope.go:122] "RemoveContainer" containerID="58a482dc2a7b39bc21d85ec8d83841d6078e21dc8f1275606a9dec50c75dd354"
Nov 03 18:13:00.147915 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.147429    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"
Nov 03 18:13:00.319714 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.319646    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"]
Nov 03 18:13:00.369775 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.369722    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62aead-5c76bd56c8-p88wc"]
Nov 03 18:13:00.821669 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.821627    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:00.980121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:00.980085    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:01.183471 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:01.183431    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:01.506857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:01.506756    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:02.122278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:02.122240    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041f2c85-51cd-40e8-9931-625774cbe220" path="/var/lib/kubelet/pods/041f2c85-51cd-40e8-9931-625774cbe220/volumes"
Nov 03 18:13:02.195313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:02.195269    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:04.321087 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.321046    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"]
Nov 03 18:13:04.321962 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.321933    5838 cpu_manager.go:414] "containerMap: removing container" podUID="041f2c85-51cd-40e8-9931-625774cbe220" containerName="maas-api"
Nov 03 18:13:04.321962 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.321959    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041f2c85-51cd-40e8-9931-625774cbe220" containerName="maas-api"
Nov 03 18:13:04.322184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.322169    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="041f2c85-51cd-40e8-9931-625774cbe220" containerName="maas-api"
Nov 03 18:13:04.327108 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.327082    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.433980    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434046    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kl2\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434088    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434151    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434192    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434277    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434353    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434376    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.434460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.434453    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.435556 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.435529    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"]
Nov 03 18:13:04.535790 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.535741    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kl2\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.535811    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.535870    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.535915    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536004    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536100    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536126    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536211    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536289    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536350    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536493    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536615 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536577    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.536717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.536694    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.537103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.537066    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.538814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.538708    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.539029 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.539009    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.599596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.599516    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.599760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.599629    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kl2\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.642921 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.642881    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:04.946038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:04.946007    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:05.081275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.081247    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"]
Nov 03 18:13:05.082485 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:13:05.082447    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40b6751_bcec_4589_91b5_0b058c080980.slice/crio-53363035592b03aace8d4dffcdebc722fe1745d603d6d6e5e52afe65a76bd896 WatchSource:0}: Error finding container 53363035592b03aace8d4dffcdebc722fe1745d603d6d6e5e52afe65a76bd896: Status 404 returned error can't find the container with id 53363035592b03aace8d4dffcdebc722fe1745d603d6d6e5e52afe65a76bd896
Nov 03 18:13:05.085177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.085138    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:13:05.085258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.085210    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:13:05.085258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.085239    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:13:05.190479 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.190443    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" event={"ID":"d40b6751-bcec-4589-91b5-0b058c080980","Type":"ContainerStarted","Data":"ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec"}
Nov 03 18:13:05.190659 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.190509    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" event={"ID":"d40b6751-bcec-4589-91b5-0b058c080980","Type":"ContainerStarted","Data":"53363035592b03aace8d4dffcdebc722fe1745d603d6d6e5e52afe65a76bd896"}
Nov 03 18:13:05.269851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.269747    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:05.314766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.314410    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podStartSLOduration=1.314389218 podStartE2EDuration="1.314389218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:05.313521359 +0000 UTC m=+2321.324259458" watchObservedRunningTime="2026-11-03 18:13:05.314389218 +0000 UTC m=+2321.325127289"
Nov 03 18:13:05.643337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:05.643300    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:06.231688 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:06.231644    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:06.645507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:06.645475    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:13:06.645978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:06.645600    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 18:13:07.644356 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:07.644318    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:13:07.644601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:07.644387    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 18:13:08.643543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:08.643494    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:13:08.644065 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:08.643550    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 18:13:09.643376 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:09.643335    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:13:09.643580 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:09.643400    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 18:13:10.330765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:10.313266    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:10.376683 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:13:10.376646    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:13:10.498561 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:10.498506    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:10.643515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:10.643373    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:13:10.643515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:10.643443    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 18:13:10.745351 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:10.745317    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:11.248466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:11.248415    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:11.403216 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:11.403176    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:11.802773 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:11.802721    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:11.803325 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:11.803311    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:11.804225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:11.804204    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:13:16.243395 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:16.243352    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"]
Nov 03 18:13:16.279508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:16.279469    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"]
Nov 03 18:13:16.374012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:16.373971    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:17.000581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:17.000532    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:17.207158 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:17.207105    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:17.361432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:17.361340    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:22.640192 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:22.640153    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:26.397963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:26.397927    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:26.556051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:26.556009    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:26.868864 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:26.868767    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:27.116761 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:27.116716    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"]
Nov 03 18:13:27.488040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:27.488006    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:27.646568 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:27.646531    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:28.207057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:28.207019    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:31.299399 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.299351    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main" containerID="cri-o://3801393122e782ff6d633a1da76a3064150b9f1affac458719b918d3f091204b" gracePeriod=45
Nov 03 18:13:31.311210 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.311171    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main" containerID="cri-o://0f30f81f0d04dcbba81864816e775abdce4a40f819e97e3ce6f05b1c4a0ae418" gracePeriod=45
Nov 03 18:13:31.510090 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.509833    5838 generic.go:356] "Generic (PLEG): container finished" podID="8e82a454-4d93-496d-8557-2d434bf780f9" containerID="0f30f81f0d04dcbba81864816e775abdce4a40f819e97e3ce6f05b1c4a0ae418" exitCode=0
Nov 03 18:13:31.510090 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.510016    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" event={"ID":"8e82a454-4d93-496d-8557-2d434bf780f9","Type":"ContainerDied","Data":"0f30f81f0d04dcbba81864816e775abdce4a40f819e97e3ce6f05b1c4a0ae418"}
Nov 03 18:13:31.512448 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.512427    5838 generic.go:356] "Generic (PLEG): container finished" podID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerID="3801393122e782ff6d633a1da76a3064150b9f1affac458719b918d3f091204b" exitCode=0
Nov 03 18:13:31.512663 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.512556    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" event={"ID":"706a5feb-cddf-4190-89a3-75f44cd6e13d","Type":"ContainerDied","Data":"3801393122e782ff6d633a1da76a3064150b9f1affac458719b918d3f091204b"}
Nov 03 18:13:31.630005 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.629973    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 18:13:31.635473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.635448    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 18:13:31.795788 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.795685    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache\") pod \"8e82a454-4d93-496d-8557-2d434bf780f9\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") "
Nov 03 18:13:31.795788 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.795783    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache\") pod \"706a5feb-cddf-4190-89a3-75f44cd6e13d\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") "
Nov 03 18:13:31.796089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.795871    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp\" (UniqueName: \"kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp\") pod \"8e82a454-4d93-496d-8557-2d434bf780f9\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") "
Nov 03 18:13:31.796089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.795908    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz\" (UniqueName: \"kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz\") pod \"706a5feb-cddf-4190-89a3-75f44cd6e13d\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") "
Nov 03 18:13:31.796089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.795949    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home\") pod \"706a5feb-cddf-4190-89a3-75f44cd6e13d\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") "
Nov 03 18:13:31.796089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.795981    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs\") pod \"8e82a454-4d93-496d-8557-2d434bf780f9\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") "
Nov 03 18:13:31.796089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796033    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm\") pod \"8e82a454-4d93-496d-8557-2d434bf780f9\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") "
Nov 03 18:13:31.796089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796059    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache" pod "706a5feb-cddf-4190-89a3-75f44cd6e13d" (UID: "706a5feb-cddf-4190-89a3-75f44cd6e13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.796477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796102    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir\") pod \"706a5feb-cddf-4190-89a3-75f44cd6e13d\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") "
Nov 03 18:13:31.796477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796167    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm\") pod \"706a5feb-cddf-4190-89a3-75f44cd6e13d\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") "
Nov 03 18:13:31.796477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796206    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home\") pod \"8e82a454-4d93-496d-8557-2d434bf780f9\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") "
Nov 03 18:13:31.796477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796283    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs\" (UniqueName: \"kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs\") pod \"706a5feb-cddf-4190-89a3-75f44cd6e13d\" (UID: \"706a5feb-cddf-4190-89a3-75f44cd6e13d\") "
Nov 03 18:13:31.796477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796318    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir\") pod \"8e82a454-4d93-496d-8557-2d434bf780f9\" (UID: \"8e82a454-4d93-496d-8557-2d434bf780f9\") "
Nov 03 18:13:31.796822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.796795    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache" pod "8e82a454-4d93-496d-8557-2d434bf780f9" (UID: "8e82a454-4d93-496d-8557-2d434bf780f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.797051    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.797072    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.797061    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir" pod "8e82a454-4d93-496d-8557-2d434bf780f9" (UID: "8e82a454-4d93-496d-8557-2d434bf780f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.797236    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home" pod "8e82a454-4d93-496d-8557-2d434bf780f9" (UID: "8e82a454-4d93-496d-8557-2d434bf780f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.797757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.797524    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home" pod "706a5feb-cddf-4190-89a3-75f44cd6e13d" (UID: "706a5feb-cddf-4190-89a3-75f44cd6e13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.798249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.798207    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir" pod "706a5feb-cddf-4190-89a3-75f44cd6e13d" (UID: "706a5feb-cddf-4190-89a3-75f44cd6e13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.819652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.800221    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp" pod "8e82a454-4d93-496d-8557-2d434bf780f9" (UID: "8e82a454-4d93-496d-8557-2d434bf780f9"). InnerVolumeSpecName "kube-api-access-7vgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:31.819652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.800622    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz" pod "706a5feb-cddf-4190-89a3-75f44cd6e13d" (UID: "706a5feb-cddf-4190-89a3-75f44cd6e13d"). InnerVolumeSpecName "kube-api-access-7zhlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:31.819652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.800985    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs" pod "706a5feb-cddf-4190-89a3-75f44cd6e13d" (UID: "706a5feb-cddf-4190-89a3-75f44cd6e13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:31.819652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.806344    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm" pod "706a5feb-cddf-4190-89a3-75f44cd6e13d" (UID: "706a5feb-cddf-4190-89a3-75f44cd6e13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.819652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.817975    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs" pod "8e82a454-4d93-496d-8557-2d434bf780f9" (UID: "8e82a454-4d93-496d-8557-2d434bf780f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:31.819652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.818355    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm" pod "8e82a454-4d93-496d-8557-2d434bf780f9" (UID: "8e82a454-4d93-496d-8557-2d434bf780f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.898362 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898320    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898362 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898360    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898372    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898384    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898395    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706a5feb-cddf-4190-89a3-75f44cd6e13d-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898417    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82a454-4d93-496d-8557-2d434bf780f9-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898430    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgzp\" (UniqueName: \"kubernetes.io/projected/8e82a454-4d93-496d-8557-2d434bf780f9-kube-api-access-7vgzp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898443    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhlz\" (UniqueName: \"kubernetes.io/projected/706a5feb-cddf-4190-89a3-75f44cd6e13d-kube-api-access-7zhlz\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898455    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706a5feb-cddf-4190-89a3-75f44cd6e13d-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.898606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:31.898468    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82a454-4d93-496d-8557-2d434bf780f9-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:32.117323 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.117286    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerName="payload-processing" containerID="cri-o://81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c" gracePeriod=25
Nov 03 18:13:32.524948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.524862    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"
Nov 03 18:13:32.524948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.524881    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r" event={"ID":"706a5feb-cddf-4190-89a3-75f44cd6e13d","Type":"ContainerDied","Data":"d0ba7ed1df4b99a0dc83194a03873826aed65411e80f0119f45d5151db533651"}
Nov 03 18:13:32.525476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.524948    5838 scope.go:122] "RemoveContainer" containerID="3801393122e782ff6d633a1da76a3064150b9f1affac458719b918d3f091204b"
Nov 03 18:13:32.526913 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.526877    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"
Nov 03 18:13:32.527402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.526879    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m" event={"ID":"8e82a454-4d93-496d-8557-2d434bf780f9","Type":"ContainerDied","Data":"e5df06eaf4e0e8a53a733147c12f599a89beb0572fb02673ef3f5bda38b20a21"}
Nov 03 18:13:32.538794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.538765    5838 scope.go:122] "RemoveContainer" containerID="0f30f81f0d04dcbba81864816e775abdce4a40f819e97e3ce6f05b1c4a0ae418"
Nov 03 18:13:32.695326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.695280    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"]
Nov 03 18:13:32.745611 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.745577    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/premium-w0-07f297-kserve-ddf4b9fbf-kzm8m"]
Nov 03 18:13:32.816468 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.816435    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"]
Nov 03 18:13:32.822449 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.822417    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-w0-07f297-kserve-5cb5886948-59h2r"]
Nov 03 18:13:32.846280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.846241    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"]
Nov 03 18:13:32.846524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.846494    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" podUID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" containerName="maas-api" containerID="cri-o://c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f" gracePeriod=30
Nov 03 18:13:32.942056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.942022    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:32.957420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:32.957326    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" podUID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.126:8443/health\": dial tcp 10.134.0.126:8443: connect: connection refused"
Nov 03 18:13:33.178403 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.178371    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:13:33.318653 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.318571    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca\") pod \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") "
Nov 03 18:13:33.318903 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.318713    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv\" (UniqueName: \"kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv\") pod \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") "
Nov 03 18:13:33.318903 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.318841    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs\") pod \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") "
Nov 03 18:13:33.319018 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.318923    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\") pod \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\" (UID: \"bc7e6943-1d9a-49c3-a89f-91110e6093c2\") "
Nov 03 18:13:33.319018 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.319005    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca" pod "bc7e6943-1d9a-49c3-a89f-91110e6093c2" (UID: "bc7e6943-1d9a-49c3-a89f-91110e6093c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:33.319480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.319458    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7e6943-1d9a-49c3-a89f-91110e6093c2-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:33.321663 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.321629    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls" pod "bc7e6943-1d9a-49c3-a89f-91110e6093c2" (UID: "bc7e6943-1d9a-49c3-a89f-91110e6093c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:33.321891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.321866    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv" pod "bc7e6943-1d9a-49c3-a89f-91110e6093c2" (UID: "bc7e6943-1d9a-49c3-a89f-91110e6093c2"). InnerVolumeSpecName "kube-api-access-9bjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:33.322749 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.322704    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs" pod "bc7e6943-1d9a-49c3-a89f-91110e6093c2" (UID: "bc7e6943-1d9a-49c3-a89f-91110e6093c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:33.440046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.439219    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjwv\" (UniqueName: \"kubernetes.io/projected/bc7e6943-1d9a-49c3-a89f-91110e6093c2-kube-api-access-9bjwv\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:33.440046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.439264    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:33.440046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.439280    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7e6943-1d9a-49c3-a89f-91110e6093c2-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:33.537621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.537538    5838 generic.go:356] "Generic (PLEG): container finished" podID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" containerID="c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f" exitCode=0
Nov 03 18:13:33.537621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.537588    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"
Nov 03 18:13:33.537621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.537613    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" event={"ID":"bc7e6943-1d9a-49c3-a89f-91110e6093c2","Type":"ContainerDied","Data":"c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f"}
Nov 03 18:13:33.538205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.537689    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p" event={"ID":"bc7e6943-1d9a-49c3-a89f-91110e6093c2","Type":"ContainerDied","Data":"48f9a7933dd6e600479c2e29651c8db939babfc7f26b71229da4f602e3ce5ab0"}
Nov 03 18:13:33.538205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.537714    5838 scope.go:122] "RemoveContainer" containerID="c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f"
Nov 03 18:13:33.549385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.549364    5838 scope.go:122] "RemoveContainer" containerID="c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f"
Nov 03 18:13:33.549693 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:13:33.549671    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f\": container with ID starting with c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f not found: ID does not exist" containerID="c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f"
Nov 03 18:13:33.549803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.549701    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f"} err="failed to get container status \"c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f\": rpc error: code = NotFound desc = could not find container \"c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f\": container with ID starting with c534418947162b1dcced55bef06b1009ce252e1f4c9dfdb771c308ae18b6780f not found: ID does not exist"
Nov 03 18:13:33.708972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.708935    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"]
Nov 03 18:13:33.714587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:33.714559    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-528e53-744b6bfd97-nwf9p"]
Nov 03 18:13:34.117671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:34.117577    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" path="/var/lib/kubelet/pods/706a5feb-cddf-4190-89a3-75f44cd6e13d/volumes"
Nov 03 18:13:34.118121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:34.118098    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" path="/var/lib/kubelet/pods/8e82a454-4d93-496d-8557-2d434bf780f9/volumes"
Nov 03 18:13:34.118555 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:34.118535    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" path="/var/lib/kubelet/pods/bc7e6943-1d9a-49c3-a89f-91110e6093c2/volumes"
Nov 03 18:13:34.511057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:34.511018    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:34.712425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:34.712336    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:34.898463 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:34.898409    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:35.049851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:35.049814    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:35.677798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:35.677720    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:39.454448 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.454405    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"]
Nov 03 18:13:39.454985 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.454710    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" podUID="c0022528-80fa-445f-8ff6-6b40803cb91d" containerName="maas-api" containerID="cri-o://cadd1d483b87974e46b46601fd87849db98653b1c1a191ac361c49c42977fdcf" gracePeriod=30
Nov 03 18:13:39.567891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.567851    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:39.587938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.587908    5838 generic.go:356] "Generic (PLEG): container finished" podID="c0022528-80fa-445f-8ff6-6b40803cb91d" containerID="cadd1d483b87974e46b46601fd87849db98653b1c1a191ac361c49c42977fdcf" exitCode=0
Nov 03 18:13:39.588109 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.587990    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" event={"ID":"c0022528-80fa-445f-8ff6-6b40803cb91d","Type":"ContainerDied","Data":"cadd1d483b87974e46b46601fd87849db98653b1c1a191ac361c49c42977fdcf"}
Nov 03 18:13:39.761476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.761441    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:13:39.820262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.820222    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\") pod \"c0022528-80fa-445f-8ff6-6b40803cb91d\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") "
Nov 03 18:13:39.820436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.820347    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8\" (UniqueName: \"kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8\") pod \"c0022528-80fa-445f-8ff6-6b40803cb91d\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") "
Nov 03 18:13:39.820436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.820412    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca\") pod \"c0022528-80fa-445f-8ff6-6b40803cb91d\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") "
Nov 03 18:13:39.820586 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.820524    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs\") pod \"c0022528-80fa-445f-8ff6-6b40803cb91d\" (UID: \"c0022528-80fa-445f-8ff6-6b40803cb91d\") "
Nov 03 18:13:39.821307 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.821225    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca" pod "c0022528-80fa-445f-8ff6-6b40803cb91d" (UID: "c0022528-80fa-445f-8ff6-6b40803cb91d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:39.822707 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.822685    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls" pod "c0022528-80fa-445f-8ff6-6b40803cb91d" (UID: "c0022528-80fa-445f-8ff6-6b40803cb91d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:39.823119 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.823102    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8" pod "c0022528-80fa-445f-8ff6-6b40803cb91d" (UID: "c0022528-80fa-445f-8ff6-6b40803cb91d"). InnerVolumeSpecName "kube-api-access-5kwx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:39.823189 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.823178    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs" pod "c0022528-80fa-445f-8ff6-6b40803cb91d" (UID: "c0022528-80fa-445f-8ff6-6b40803cb91d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:39.922967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.922923    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:39.922967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.922968    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kwx8\" (UniqueName: \"kubernetes.io/projected/c0022528-80fa-445f-8ff6-6b40803cb91d-kube-api-access-5kwx8\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:39.923193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.922989    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0022528-80fa-445f-8ff6-6b40803cb91d-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:39.923193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:39.923003    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0022528-80fa-445f-8ff6-6b40803cb91d-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:40.340496 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:13:40.340452    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:13:40.596689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:40.596647    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"
Nov 03 18:13:40.596689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:40.596680    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq" event={"ID":"c0022528-80fa-445f-8ff6-6b40803cb91d","Type":"ContainerDied","Data":"3f5b0d1f9cb41c123d8a15cdcd88e7f70578e982caf99c396088c442b7df88d3"}
Nov 03 18:13:40.597297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:40.596825    5838 scope.go:122] "RemoveContainer" containerID="cadd1d483b87974e46b46601fd87849db98653b1c1a191ac361c49c42977fdcf"
Nov 03 18:13:40.786841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:40.786789    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"]
Nov 03 18:13:40.792314 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:40.792279    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b05459-77f86db4c6-s8jjq"]
Nov 03 18:13:40.903525 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:40.903481    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:41.059102 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:41.059060    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:41.213980 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:41.213890    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:41.386690 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:41.386652    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:41.494051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:41.493962    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:41.540963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:41.540919    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:42.119426 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:42.119390    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0022528-80fa-445f-8ff6-6b40803cb91d" path="/var/lib/kubelet/pods/c0022528-80fa-445f-8ff6-6b40803cb91d/volumes"
Nov 03 18:13:42.211302 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:42.211252    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:43.781574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:43.781537    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:45.001554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:45.001427    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:45.189480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:45.189441    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:45.785154 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:45.782863    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:46.148172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:46.148139    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:46.902851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:46.902809    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:49.095492 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:49.095460    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:49.248519 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:49.248480    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:49.815915 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:49.815875    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:49.970871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:49.970832    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:50.128778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:50.128673    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:50.292042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:50.291998    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:51.492674 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:51.492627    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:52.227694 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.227657    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:52.413009 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.412977    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:13:52.414941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.414914    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 18:13:52.481365 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.481332    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2\" (UniqueName: \"kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2\") pod \"936f2578-8d84-4dab-b254-814e0fdc6ef0\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") "
Nov 03 18:13:52.481570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.481549    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume\") pod \"936f2578-8d84-4dab-b254-814e0fdc6ef0\" (UID: \"936f2578-8d84-4dab-b254-814e0fdc6ef0\") "
Nov 03 18:13:52.482007 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.481979    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume" pod "936f2578-8d84-4dab-b254-814e0fdc6ef0" (UID: "936f2578-8d84-4dab-b254-814e0fdc6ef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:52.482271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.482235    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936f2578-8d84-4dab-b254-814e0fdc6ef0-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:52.484130 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.484102    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2" pod "936f2578-8d84-4dab-b254-814e0fdc6ef0" (UID: "936f2578-8d84-4dab-b254-814e0fdc6ef0"). InnerVolumeSpecName "kube-api-access-hmpx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:52.583249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.583213    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmpx2\" (UniqueName: \"kubernetes.io/projected/936f2578-8d84-4dab-b254-814e0fdc6ef0-kube-api-access-hmpx2\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:52.694461 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.694368    5838 generic.go:356] "Generic (PLEG): container finished" podID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerID="81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c" exitCode=0
Nov 03 18:13:52.694461 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.694414    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"
Nov 03 18:13:52.694461 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.694447    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" event={"ID":"936f2578-8d84-4dab-b254-814e0fdc6ef0","Type":"ContainerDied","Data":"81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c"}
Nov 03 18:13:52.694700 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.694488    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4" event={"ID":"936f2578-8d84-4dab-b254-814e0fdc6ef0","Type":"ContainerDied","Data":"5ae7534c76c724a303d2d5a7f4ccea283fe5c330c2e38d16dc3f6b522ec3cbf5"}
Nov 03 18:13:52.694700 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.694510    5838 scope.go:122] "RemoveContainer" containerID="81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c"
Nov 03 18:13:52.706177 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.706152    5838 scope.go:122] "RemoveContainer" containerID="81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c"
Nov 03 18:13:52.706496 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:13:52.706470    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c\": container with ID starting with 81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c not found: ID does not exist" containerID="81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c"
Nov 03 18:13:52.706613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.706505    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c"} err="failed to get container status \"81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c\": rpc error: code = NotFound desc = could not find container \"81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c\": container with ID starting with 81c07f1f11162f0f2748ef52c57b65fb43aa9b3deacf31ce3ce96ebf7383962c not found: ID does not exist"
Nov 03 18:13:52.864328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.864282    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"]
Nov 03 18:13:52.873983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:52.873951    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-07f297-54bb76975c-2rvt4"]
Nov 03 18:13:53.621894 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:53.621851    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:53.788876 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:53.788814    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:53.943370 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:53.943329    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:54.121413 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:54.121367    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" path="/var/lib/kubelet/pods/936f2578-8d84-4dab-b254-814e0fdc6ef0/volumes"
Nov 03 18:13:54.476655 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:54.476618    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:55.235614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:55.235582    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:55.980996 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:55.980963    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:57.622003 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:57.621967    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:58.026399 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:58.026339    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:58.182919 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:58.182870    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:58.337590 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:13:58.337481    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:02.412456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:02.412380    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:02.567621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:02.567578    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:03.679196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:03.679154    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:05.047910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:05.047876    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:05.201703 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:05.201648    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:08.863173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:08.863129    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:09.026161 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:09.026126    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:09.187565 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:09.187525    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:10.341930 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:10.341882    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:14:13.882936 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:13.882902    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"]
Nov 03 18:14:13.883429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:13.883158    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" podUID="68411659-153d-4907-90d4-4e34648af4f5" containerName="maas-api" containerID="cri-o://8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a" gracePeriod=30
Nov 03 18:14:13.982696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:13.982649    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:14.182562 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.182528    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:14:14.320935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.320899    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\") pod \"68411659-153d-4907-90d4-4e34648af4f5\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") "
Nov 03 18:14:14.321158 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.321140    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca\") pod \"68411659-153d-4907-90d4-4e34648af4f5\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") "
Nov 03 18:14:14.321239 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.321196    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc\" (UniqueName: \"kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc\") pod \"68411659-153d-4907-90d4-4e34648af4f5\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") "
Nov 03 18:14:14.321304 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.321269    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs\") pod \"68411659-153d-4907-90d4-4e34648af4f5\" (UID: \"68411659-153d-4907-90d4-4e34648af4f5\") "
Nov 03 18:14:14.321917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.321540    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca" pod "68411659-153d-4907-90d4-4e34648af4f5" (UID: "68411659-153d-4907-90d4-4e34648af4f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:14.321917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.321878    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68411659-153d-4907-90d4-4e34648af4f5-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:14.324159 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.324123    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc" pod "68411659-153d-4907-90d4-4e34648af4f5" (UID: "68411659-153d-4907-90d4-4e34648af4f5"). InnerVolumeSpecName "kube-api-access-l5vfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:14.324280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.324171    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls" pod "68411659-153d-4907-90d4-4e34648af4f5" (UID: "68411659-153d-4907-90d4-4e34648af4f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:14.324905 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.324878    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs" pod "68411659-153d-4907-90d4-4e34648af4f5" (UID: "68411659-153d-4907-90d4-4e34648af4f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:14.423258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.423155    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5vfc\" (UniqueName: \"kubernetes.io/projected/68411659-153d-4907-90d4-4e34648af4f5-kube-api-access-l5vfc\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:14.423258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.423205    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:14.423258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.423218    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68411659-153d-4907-90d4-4e34648af4f5-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:14.895781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.895711    5838 generic.go:356] "Generic (PLEG): container finished" podID="68411659-153d-4907-90d4-4e34648af4f5" containerID="8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a" exitCode=0
Nov 03 18:14:14.896760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.896711    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" event={"ID":"68411659-153d-4907-90d4-4e34648af4f5","Type":"ContainerDied","Data":"8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a"}
Nov 03 18:14:14.896960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.896934    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq" event={"ID":"68411659-153d-4907-90d4-4e34648af4f5","Type":"ContainerDied","Data":"25a8585cd28cae1d97d924e67796c5bb68b37be97dd2b89d146dc01df691d290"}
Nov 03 18:14:14.897080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.897068    5838 scope.go:122] "RemoveContainer" containerID="8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a"
Nov 03 18:14:14.897314 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.897301    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"
Nov 03 18:14:14.937797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.937762    5838 scope.go:122] "RemoveContainer" containerID="8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a"
Nov 03 18:14:14.938266 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:14.938236    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a\": container with ID starting with 8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a not found: ID does not exist" containerID="8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a"
Nov 03 18:14:14.938383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:14.938280    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a"} err="failed to get container status \"8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a\": rpc error: code = NotFound desc = could not find container \"8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a\": container with ID starting with 8b47d4cf08a54e4de8ffd3a059a0098af4e5be28558c82079bc99a471473278a not found: ID does not exist"
Nov 03 18:14:15.073057 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:15.073021    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"]
Nov 03 18:14:15.123387 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:15.123351    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ed3e95c-655698f9dd-428fq"]
Nov 03 18:14:15.669378 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:15.669342    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:15.933798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:15.933703    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:16.085310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:16.085277    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:16.118836 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:16.118799    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68411659-153d-4907-90d4-4e34648af4f5" path="/var/lib/kubelet/pods/68411659-153d-4907-90d4-4e34648af4f5/volumes"
Nov 03 18:14:16.614959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:16.614893    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:21.677778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:21.677689    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:21.678813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:21.678770    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"]
Nov 03 18:14:21.832300 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:21.832269    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:24.648806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:24.648779    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:14:24.654219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:24.654190    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:14:25.102119 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:25.102038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:25.577853 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:25.577813    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:26.390438 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.390396    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"]
Nov 03 18:14:26.390884 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.390636    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" podUID="ece16545-f72f-45ac-bfae-2723829f1d31" containerName="maas-api" containerID="cri-o://41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108" gracePeriod=30
Nov 03 18:14:26.506137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.506045    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:26.666927 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.666844    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:26.732587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.732557    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 18:14:26.867386 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.867272    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:26.929081 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.928991    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca\") pod \"ece16545-f72f-45ac-bfae-2723829f1d31\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") "
Nov 03 18:14:26.929081 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.929034    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8\" (UniqueName: \"kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8\") pod \"ece16545-f72f-45ac-bfae-2723829f1d31\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") "
Nov 03 18:14:26.929313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.929161    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs\") pod \"ece16545-f72f-45ac-bfae-2723829f1d31\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") "
Nov 03 18:14:26.929383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.929332    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\") pod \"ece16545-f72f-45ac-bfae-2723829f1d31\" (UID: \"ece16545-f72f-45ac-bfae-2723829f1d31\") "
Nov 03 18:14:26.929483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.929453    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca" pod "ece16545-f72f-45ac-bfae-2723829f1d31" (UID: "ece16545-f72f-45ac-bfae-2723829f1d31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:26.929916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.929892    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece16545-f72f-45ac-bfae-2723829f1d31-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:26.932045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.931989    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8" pod "ece16545-f72f-45ac-bfae-2723829f1d31" (UID: "ece16545-f72f-45ac-bfae-2723829f1d31"). InnerVolumeSpecName "kube-api-access-cftd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:26.932168 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.932086    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls" pod "ece16545-f72f-45ac-bfae-2723829f1d31" (UID: "ece16545-f72f-45ac-bfae-2723829f1d31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:26.932289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:26.932265    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs" pod "ece16545-f72f-45ac-bfae-2723829f1d31" (UID: "ece16545-f72f-45ac-bfae-2723829f1d31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:27.007046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.007004    5838 generic.go:356] "Generic (PLEG): container finished" podID="ece16545-f72f-45ac-bfae-2723829f1d31" containerID="41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108" exitCode=0
Nov 03 18:14:27.007266 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.007070    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"
Nov 03 18:14:27.007266 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.007095    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" event={"ID":"ece16545-f72f-45ac-bfae-2723829f1d31","Type":"ContainerDied","Data":"41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108"}
Nov 03 18:14:27.007266 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.007161    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt" event={"ID":"ece16545-f72f-45ac-bfae-2723829f1d31","Type":"ContainerDied","Data":"9b089884d7f8f0b0b91013c27b8c21b4e216e15421188f9b9a5ac475ab7d22b5"}
Nov 03 18:14:27.007266 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.007191    5838 scope.go:122] "RemoveContainer" containerID="41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108"
Nov 03 18:14:27.021843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.021813    5838 scope.go:122] "RemoveContainer" containerID="41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108"
Nov 03 18:14:27.022248 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:27.022221    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108\": container with ID starting with 41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108 not found: ID does not exist" containerID="41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108"
Nov 03 18:14:27.022326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.022261    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108"} err="failed to get container status \"41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108\": rpc error: code = NotFound desc = could not find container \"41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108\": container with ID starting with 41ffba72b0ef737c7324ca8c2aaa4ae6aa1cf8c2e02f3dece8843204c5141108 not found: ID does not exist"
Nov 03 18:14:27.031626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.031596    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.031626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.031626    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cftd8\" (UniqueName: \"kubernetes.io/projected/ece16545-f72f-45ac-bfae-2723829f1d31-kube-api-access-cftd8\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.031835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.031642    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece16545-f72f-45ac-bfae-2723829f1d31-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.177035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.176990    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"]
Nov 03 18:14:27.183243 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:27.183163    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-07f297-789b84544-j6jqt"]
Nov 03 18:14:28.119666 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:28.119628    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece16545-f72f-45ac-bfae-2723829f1d31" path="/var/lib/kubelet/pods/ece16545-f72f-45ac-bfae-2723829f1d31/volumes"
Nov 03 18:14:28.210067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:28.210033    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:28.541978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:28.541944    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:28.699440 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:28.699406    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:29.293452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:29.293420    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:31.313833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.313794    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"]
Nov 03 18:14:31.314405 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314390    5838 cpu_manager.go:414] "containerMap: removing container" podUID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" containerName="maas-api"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314406    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" containerName="maas-api"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314429    5838 cpu_manager.go:414] "containerMap: removing container" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314434    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314449    5838 cpu_manager.go:414] "containerMap: removing container" podUID="68411659-153d-4907-90d4-4e34648af4f5" containerName="maas-api"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314454    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68411659-153d-4907-90d4-4e34648af4f5" containerName="maas-api"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314465    5838 cpu_manager.go:414] "containerMap: removing container" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main"
Nov 03 18:14:31.314473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314470    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314489    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c0022528-80fa-445f-8ff6-6b40803cb91d" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314494    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0022528-80fa-445f-8ff6-6b40803cb91d" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314501    5838 cpu_manager.go:414] "containerMap: removing container" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerName="payload-processing"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314506    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerName="payload-processing"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314515    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ece16545-f72f-45ac-bfae-2723829f1d31" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314520    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece16545-f72f-45ac-bfae-2723829f1d31" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314606    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="68411659-153d-4907-90d4-4e34648af4f5" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314618    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e82a454-4d93-496d-8557-2d434bf780f9" containerName="main"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314628    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0022528-80fa-445f-8ff6-6b40803cb91d" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314635    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="936f2578-8d84-4dab-b254-814e0fdc6ef0" containerName="payload-processing"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314641    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece16545-f72f-45ac-bfae-2723829f1d31" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314648    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7e6943-1d9a-49c3-a89f-91110e6093c2" containerName="maas-api"
Nov 03 18:14:31.314841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.314656    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="706a5feb-cddf-4190-89a3-75f44cd6e13d" containerName="main"
Nov 03 18:14:31.319821 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.319798    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390519 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390480    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390519 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390520    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390638    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390661    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn27n\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390680    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390933 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390805    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.390933 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390918    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.391030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390946    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.391030 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.390966    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.429142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.429104    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"]
Nov 03 18:14:31.492355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492317    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492355 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492355    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492379    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492460    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492479    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492690 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492661    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492723    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn27n\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492790    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492847 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492836    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492844    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492904    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.492949 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.492922    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.493239 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.493203    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.493360 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.493336    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.495104 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.495082    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.495509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.495488    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.556066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.556032    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.556316 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.556293    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn27n\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.637845 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.637630    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:31.656982 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.656908    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:31.857500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:31.844088    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:32.004645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:32.004608    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:32.081002 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:32.079493    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"]
Nov 03 18:14:32.081514 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:14:32.081468    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04be7051_223e_4d83_9f57_fcefdc8470b2.slice/crio-3d6cd1dd52d610bc6f18dbcee8ceed723b89bdd039006d020e82f7a8a50bfbbd WatchSource:0}: Error finding container 3d6cd1dd52d610bc6f18dbcee8ceed723b89bdd039006d020e82f7a8a50bfbbd: Status 404 returned error can't find the container with id 3d6cd1dd52d610bc6f18dbcee8ceed723b89bdd039006d020e82f7a8a50bfbbd
Nov 03 18:14:32.084646 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:32.084609    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:14:32.084766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:32.084680    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:14:32.084766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:32.084713    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:14:33.057305 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:33.057264    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" event={"ID":"04be7051-223e-4d83-9f57-fcefdc8470b2","Type":"ContainerStarted","Data":"2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba"}
Nov 03 18:14:33.057844 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:33.057818    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" event={"ID":"04be7051-223e-4d83-9f57-fcefdc8470b2","Type":"ContainerStarted","Data":"3d6cd1dd52d610bc6f18dbcee8ceed723b89bdd039006d020e82f7a8a50bfbbd"}
Nov 03 18:14:33.285752 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:33.285679    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podStartSLOduration=2.285664225 podStartE2EDuration="2.285664225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:33.234015512 +0000 UTC m=+2409.244753603" watchObservedRunningTime="2026-11-03 18:14:33.285664225 +0000 UTC m=+2409.296402296"
Nov 03 18:14:33.638828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:33.638785    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:33.640719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:33.640691    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 18:14:33.640880 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:33.640767    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 18:14:34.081807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:34.081698    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:34.638249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:34.638203    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 18:14:34.638486 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:34.638272    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 18:14:35.010758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.007722    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:35.277043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.276165    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:35.416996 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.416958    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"]
Nov 03 18:14:35.435176 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.435141    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:14:35.435473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.435448    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.475223 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.475187    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"]
Nov 03 18:14:35.490600 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.490052    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0adf17\""
Nov 03 18:14:35.490600 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.490310    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0adf17-dockercfg-c6pjp\""
Nov 03 18:14:35.556916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.556802    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"]
Nov 03 18:14:35.611759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.593636    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.611759 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.606806    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48t78\" (UniqueName: \"kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78\") pod \"payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.624826 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.621395    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.667269 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.667229    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 18:14:35.667458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.667305    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 18:14:35.668310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.668283    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"]
Nov 03 18:14:35.723758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.720067    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2vp\" (UniqueName: \"kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp\") pod \"payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.723758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.720131    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.723758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.720176    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48t78\" (UniqueName: \"kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78\") pod \"payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.723758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.720267    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.723758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.721290    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.786877 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.786675    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48t78\" (UniqueName: \"kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78\") pod \"payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:35.829828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.829588    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.829828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.829805    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2vp\" (UniqueName: \"kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp\") pod \"payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.832807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.831367    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.893930 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.893898    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2vp\" (UniqueName: \"kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp\") pod \"payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:35.983069 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:35.983033    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:36.052183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:36.052146    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:36.419756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:36.419361    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"]
Nov 03 18:14:36.423572 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:14:36.423540    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a365a1_5679_46ce_9e4e_31421d540860.slice/crio-ae02ea7d34ce8bde305f9b39c38990b171acc8582296fb5d282fe47d3290e961 WatchSource:0}: Error finding container ae02ea7d34ce8bde305f9b39c38990b171acc8582296fb5d282fe47d3290e961: Status 404 returned error can't find the container with id ae02ea7d34ce8bde305f9b39c38990b171acc8582296fb5d282fe47d3290e961
Nov 03 18:14:36.532099 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:36.531903    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"]
Nov 03 18:14:36.532099 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:14:36.532051    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73e1e65_3dcf_490d_987f_2510b787e372.slice/crio-703f352d53a2c1b5b28d361b2cbb841fee3432d8c869e70dcf1e1f22aba4cac2 WatchSource:0}: Error finding container 703f352d53a2c1b5b28d361b2cbb841fee3432d8c869e70dcf1e1f22aba4cac2: Status 404 returned error can't find the container with id 703f352d53a2c1b5b28d361b2cbb841fee3432d8c869e70dcf1e1f22aba4cac2
Nov 03 18:14:36.638844 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:36.638713    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 18:14:36.638844 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:36.638795    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 18:14:36.727814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:36.719914    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerName="main" containerID="cri-o://f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6" gracePeriod=45
Nov 03 18:14:37.167492 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.167457    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:14:37.170043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.170009    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" event={"ID":"c6a365a1-5679-46ce-9e4e-31421d540860","Type":"ContainerStarted","Data":"2421720b4c818d5a14b89348d9678f062a08e86af9639388167426e21ce3ab4e"}
Nov 03 18:14:37.170191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.170063    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" event={"ID":"c6a365a1-5679-46ce-9e4e-31421d540860","Type":"ContainerStarted","Data":"ae02ea7d34ce8bde305f9b39c38990b171acc8582296fb5d282fe47d3290e961"}
Nov 03 18:14:37.170191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.170093    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:37.171840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.171817    5838 generic.go:356] "Generic (PLEG): container finished" podID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerID="f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6" exitCode=0
Nov 03 18:14:37.171958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.171860    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"
Nov 03 18:14:37.171958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.171890    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" event={"ID":"e07e2162-258d-4d72-af74-4ccd27078ef2","Type":"ContainerDied","Data":"f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6"}
Nov 03 18:14:37.171958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.171920    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" event={"ID":"e07e2162-258d-4d72-af74-4ccd27078ef2","Type":"ContainerDied","Data":"2036ba4ed5c5e3f55fd966b9ce16e44f29718b10aa8ee867eb1e3aba7681b0ba"}
Nov 03 18:14:37.171958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.171943    5838 scope.go:122] "RemoveContainer" containerID="f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6"
Nov 03 18:14:37.173875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.173842    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" event={"ID":"a73e1e65-3dcf-490d-987f-2510b787e372","Type":"ContainerStarted","Data":"17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d"}
Nov 03 18:14:37.173875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.173876    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" event={"ID":"a73e1e65-3dcf-490d-987f-2510b787e372","Type":"ContainerStarted","Data":"703f352d53a2c1b5b28d361b2cbb841fee3432d8c869e70dcf1e1f22aba4cac2"}
Nov 03 18:14:37.174112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.174091    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:37.183035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.183019    5838 scope.go:122] "RemoveContainer" containerID="f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6"
Nov 03 18:14:37.183296 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:37.183275    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6\": container with ID starting with f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6 not found: ID does not exist" containerID="f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6"
Nov 03 18:14:37.183430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.183305    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6"} err="failed to get container status \"f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6\": rpc error: code = NotFound desc = could not find container \"f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6\": container with ID starting with f7d0b8ed461f7f05900a1c1bae488a17213357e0419aaca0594227379cb6bef6 not found: ID does not exist"
Nov 03 18:14:37.263715 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.263676    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir\") pod \"e07e2162-258d-4d72-af74-4ccd27078ef2\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") "
Nov 03 18:14:37.263906 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.263797    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache\") pod \"e07e2162-258d-4d72-af74-4ccd27078ef2\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") "
Nov 03 18:14:37.263986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.263907    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx\" (UniqueName: \"kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx\") pod \"e07e2162-258d-4d72-af74-4ccd27078ef2\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") "
Nov 03 18:14:37.263986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.263967    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs\") pod \"e07e2162-258d-4d72-af74-4ccd27078ef2\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") "
Nov 03 18:14:37.264110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.263988    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir" pod "e07e2162-258d-4d72-af74-4ccd27078ef2" (UID: "e07e2162-258d-4d72-af74-4ccd27078ef2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:37.264110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.264027    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache" pod "e07e2162-258d-4d72-af74-4ccd27078ef2" (UID: "e07e2162-258d-4d72-af74-4ccd27078ef2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:37.264110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.264046    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm\") pod \"e07e2162-258d-4d72-af74-4ccd27078ef2\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") "
Nov 03 18:14:37.264110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.264095    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home\") pod \"e07e2162-258d-4d72-af74-4ccd27078ef2\" (UID: \"e07e2162-258d-4d72-af74-4ccd27078ef2\") "
Nov 03 18:14:37.264436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.264401    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home" pod "e07e2162-258d-4d72-af74-4ccd27078ef2" (UID: "e07e2162-258d-4d72-af74-4ccd27078ef2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:37.271538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.266508    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:37.271538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.266551    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:37.271538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.266567    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:37.276975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.276939    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx" pod "e07e2162-258d-4d72-af74-4ccd27078ef2" (UID: "e07e2162-258d-4d72-af74-4ccd27078ef2"). InnerVolumeSpecName "kube-api-access-vs8qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:37.277208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.277179    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm" pod "e07e2162-258d-4d72-af74-4ccd27078ef2" (UID: "e07e2162-258d-4d72-af74-4ccd27078ef2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:37.277208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.277190    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs" pod "e07e2162-258d-4d72-af74-4ccd27078ef2" (UID: "e07e2162-258d-4d72-af74-4ccd27078ef2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:37.367526 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.367454    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07e2162-258d-4d72-af74-4ccd27078ef2-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:37.367526 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.367484    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs8qx\" (UniqueName: \"kubernetes.io/projected/e07e2162-258d-4d72-af74-4ccd27078ef2-kube-api-access-vs8qx\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:37.367526 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.367499    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07e2162-258d-4d72-af74-4ccd27078ef2-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:37.460480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.460412    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" podStartSLOduration=2.460389331 podStartE2EDuration="2.460389331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:37.460179889 +0000 UTC m=+2413.470917962" watchObservedRunningTime="2026-11-03 18:14:37.460389331 +0000 UTC m=+2413.471127404"
Nov 03 18:14:37.593802 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.593659    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" podStartSLOduration=2.593636647 podStartE2EDuration="2.593636647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:37.590195345 +0000 UTC m=+2413.600933438" watchObservedRunningTime="2026-11-03 18:14:37.593636647 +0000 UTC m=+2413.604374720"
Nov 03 18:14:37.639031 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.638986    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 18:14:37.639241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.639054    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 18:14:37.731881 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.731841    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"]
Nov 03 18:14:37.750046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:37.750002    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m"]
Nov 03 18:14:38.124971 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.124929    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6ed3e95c/e2e-lifecycle-model-6ed3e95c-kserve-76f89996f-tnq4m" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:38.128982 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.128945    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" path="/var/lib/kubelet/pods/e07e2162-258d-4d72-af74-4ccd27078ef2/volumes"
Nov 03 18:14:38.638527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.638489    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 18:14:38.638970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.638560    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 18:14:38.738690 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.738657    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:38.794912 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.794869    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"]
Nov 03 18:14:38.795411 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.795381    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" podUID="f2b5f5de-cb0f-440f-98d1-fead0c207e59" containerName="maas-api" containerID="cri-o://685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9" gracePeriod=30
Nov 03 18:14:38.928271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:38.928232    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:39.097814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.097784    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"]
Nov 03 18:14:39.098022 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.097973    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy" containerID="cri-o://2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba" gracePeriod=30
Nov 03 18:14:39.103269 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.103235    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"]
Nov 03 18:14:39.103699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.103660    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" containerID="cri-o://b9382ecdb8ec619c6d05f36aa8de892e3a31f7179f573acaa71960bb6e436c11" gracePeriod=30
Nov 03 18:14:39.132483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.132456    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:39.154050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.154021    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:14:39.191155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.191053    5838 generic.go:356] "Generic (PLEG): container finished" podID="f2b5f5de-cb0f-440f-98d1-fead0c207e59" containerID="685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9" exitCode=0
Nov 03 18:14:39.191155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.191095    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"
Nov 03 18:14:39.191155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.191137    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" event={"ID":"f2b5f5de-cb0f-440f-98d1-fead0c207e59","Type":"ContainerDied","Data":"685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9"}
Nov 03 18:14:39.191466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.191202    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4" event={"ID":"f2b5f5de-cb0f-440f-98d1-fead0c207e59","Type":"ContainerDied","Data":"feb439f269b1676b1112ca968f0f1e846e5bc55ccf7268b39a25349c79768efa"}
Nov 03 18:14:39.191466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.191236    5838 scope.go:122] "RemoveContainer" containerID="685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9"
Nov 03 18:14:39.198813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.198787    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs\") pod \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") "
Nov 03 18:14:39.198957 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.198831    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4\" (UniqueName: \"kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4\") pod \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") "
Nov 03 18:14:39.198957 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.198938    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca\") pod \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") "
Nov 03 18:14:39.198957 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.198957    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\") pod \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\" (UID: \"f2b5f5de-cb0f-440f-98d1-fead0c207e59\") "
Nov 03 18:14:39.200084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.200042    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca" pod "f2b5f5de-cb0f-440f-98d1-fead0c207e59" (UID: "f2b5f5de-cb0f-440f-98d1-fead0c207e59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:39.201485 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.201451    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls" pod "f2b5f5de-cb0f-440f-98d1-fead0c207e59" (UID: "f2b5f5de-cb0f-440f-98d1-fead0c207e59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:39.201598 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.201575    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs" pod "f2b5f5de-cb0f-440f-98d1-fead0c207e59" (UID: "f2b5f5de-cb0f-440f-98d1-fead0c207e59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:39.201668 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.201620    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4" pod "f2b5f5de-cb0f-440f-98d1-fead0c207e59" (UID: "f2b5f5de-cb0f-440f-98d1-fead0c207e59"). InnerVolumeSpecName "kube-api-access-8vmm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:39.202510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.202480    5838 scope.go:122] "RemoveContainer" containerID="685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9"
Nov 03 18:14:39.202936 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:39.202903    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9\": container with ID starting with 685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9 not found: ID does not exist" containerID="685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9"
Nov 03 18:14:39.203048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.202946    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9"} err="failed to get container status \"685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9\": rpc error: code = NotFound desc = could not find container \"685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9\": container with ID starting with 685fb350cbf5aa137399a6661fd380227338030da243bb1aff48e807da5985a9 not found: ID does not exist"
Nov 03 18:14:39.302879 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.302836    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.302879 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.302878    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmm4\" (UniqueName: \"kubernetes.io/projected/f2b5f5de-cb0f-440f-98d1-fead0c207e59-kube-api-access-8vmm4\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.303171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.302894    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5f5de-cb0f-440f-98d1-fead0c207e59-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.303171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.302906    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b5f5de-cb0f-440f-98d1-fead0c207e59-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.318338 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.318292    5838 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:39.318532 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.318365    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:39.618371 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.618334    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"]
Nov 03 18:14:39.624590 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:39.624558    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-640dc9-556489c96d-tpxb4"]
Nov 03 18:14:40.116907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:40.116873    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b5f5de-cb0f-440f-98d1-fead0c207e59" path="/var/lib/kubelet/pods/f2b5f5de-cb0f-440f-98d1-fead0c207e59/volumes"
Nov 03 18:14:40.340155 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:40.340120    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:14:40.699886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:40.699853    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:40.921106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:40.921073    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:41.072103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:41.072011    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:41.633726 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:41.633683    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:44.232115 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.232084    5838 generic.go:356] "Generic (PLEG): container finished" podID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerID="b9382ecdb8ec619c6d05f36aa8de892e3a31f7179f573acaa71960bb6e436c11" exitCode=0
Nov 03 18:14:44.232540 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.232158    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" event={"ID":"a8effaaf-1861-4257-8784-3c0df65f2d30","Type":"ContainerDied","Data":"b9382ecdb8ec619c6d05f36aa8de892e3a31f7179f573acaa71960bb6e436c11"}
Nov 03 18:14:44.232540 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.232221    5838 scope.go:122] "RemoveContainer" containerID="2ef3533508218569dc87873cc682e6c93bf8019c4417aac1d1c3f72b11fc7150"
Nov 03 18:14:44.373978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.373906    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 18:14:44.462481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462444    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.462481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462493    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.462800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462526    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.462800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462595    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.462800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462658    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.462800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462694    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.462800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462724    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.463063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462839    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.463063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462876    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.463063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462901    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:44.463063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.462991    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.463063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463009    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.463298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463120    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.463298 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463131    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token\") pod \"a8effaaf-1861-4257-8784-3c0df65f2d30\" (UID: \"a8effaaf-1861-4257-8784-3c0df65f2d30\") "
Nov 03 18:14:44.463882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463855    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8effaaf-1861-4257-8784-3c0df65f2d30-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.463882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463884    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.464088 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463899    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.464088 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463913    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.464088 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.463928    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.465143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.465115    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.465345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.465156    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:44.465345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.465236    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "kube-api-access-z8nml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:44.465432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.465374    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token" pod "a8effaaf-1861-4257-8784-3c0df65f2d30" (UID: "a8effaaf-1861-4257-8784-3c0df65f2d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:44.552325 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.552287    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:44.565786 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.565726    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8nml\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-kube-api-access-z8nml\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.565786 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.565791    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.566000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.565808    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.566000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:44.565823    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8effaaf-1861-4257-8784-3c0df65f2d30-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.050758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.050713    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:45.173553 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173459    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173553 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173505    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173567    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173648    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173677    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173721    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173786    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173838    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.173857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.173859    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data\") pod \"04be7051-223e-4d83-9f57-fcefdc8470b2\" (UID: \"04be7051-223e-4d83-9f57-fcefdc8470b2\") "
Nov 03 18:14:45.174225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174010    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:45.174225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174028    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:45.174225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174090    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:45.174383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174226    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:45.174527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174503    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.174527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174523    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:45.174689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174530    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04be7051-223e-4d83-9f57-fcefdc8470b2-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.174689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174573    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.174689 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.174590    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.176514 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.176488    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:45.176743 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.176572    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:45.176988 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.176966    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:45.177056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.176997    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n" pod "04be7051-223e-4d83-9f57-fcefdc8470b2" (UID: "04be7051-223e-4d83-9f57-fcefdc8470b2"). InnerVolumeSpecName "kube-api-access-qn27n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:45.246187 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.246083    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"
Nov 03 18:14:45.246187 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.246124    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw" event={"ID":"a8effaaf-1861-4257-8784-3c0df65f2d30","Type":"ContainerDied","Data":"abe633393f6477ecff882feefe19a4c618271d92197690c3cffce1f865b5ac38"}
Nov 03 18:14:45.246871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.246842    5838 scope.go:122] "RemoveContainer" containerID="b9382ecdb8ec619c6d05f36aa8de892e3a31f7179f573acaa71960bb6e436c11"
Nov 03 18:14:45.257021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.248711    5838 generic.go:356] "Generic (PLEG): container finished" podID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerID="2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba" exitCode=0
Nov 03 18:14:45.257021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.248862    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" event={"ID":"04be7051-223e-4d83-9f57-fcefdc8470b2","Type":"ContainerDied","Data":"2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba"}
Nov 03 18:14:45.257021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.248907    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq" event={"ID":"04be7051-223e-4d83-9f57-fcefdc8470b2","Type":"ContainerDied","Data":"3d6cd1dd52d610bc6f18dbcee8ceed723b89bdd039006d020e82f7a8a50bfbbd"}
Nov 03 18:14:45.257021 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.248980    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"
Nov 03 18:14:45.262127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.262098    5838 scope.go:122] "RemoveContainer" containerID="2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba"
Nov 03 18:14:45.272432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.272397    5838 scope.go:122] "RemoveContainer" containerID="2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba"
Nov 03 18:14:45.272830 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:45.272754    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba\": container with ID starting with 2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba not found: ID does not exist" containerID="2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba"
Nov 03 18:14:45.272830 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.272795    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba"} err="failed to get container status \"2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba\": rpc error: code = NotFound desc = could not find container \"2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba\": container with ID starting with 2933a99d9da0122787659f9e10337434635692da1035e3432aed0ee41aee18ba not found: ID does not exist"
Nov 03 18:14:45.275508 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.275482    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.275614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.275510    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.275614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.275523    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.275614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.275536    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn27n\" (UniqueName: \"kubernetes.io/projected/04be7051-223e-4d83-9f57-fcefdc8470b2-kube-api-access-qn27n\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.275614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.275551    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04be7051-223e-4d83-9f57-fcefdc8470b2-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:45.416866 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.416828    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"]
Nov 03 18:14:45.422757 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.422713    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-56zzw"]
Nov 03 18:14:45.538110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.538022    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"]
Nov 03 18:14:45.546568 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.546526    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-fdfcq"]
Nov 03 18:14:45.897545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.897511    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"]
Nov 03 18:14:45.898206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898190    5838 cpu_manager.go:414] "containerMap: removing container" podUID="f2b5f5de-cb0f-440f-98d1-fead0c207e59" containerName="maas-api"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898207    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b5f5de-cb0f-440f-98d1-fead0c207e59" containerName="maas-api"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898228    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898233    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898243    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898248    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898256    5838 cpu_manager.go:414] "containerMap: removing container" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898261    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy"
Nov 03 18:14:45.898278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898278    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerName="main"
Nov 03 18:14:45.898527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898283    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerName="main"
Nov 03 18:14:45.898527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898384    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07e2162-258d-4d72-af74-4ccd27078ef2" containerName="main"
Nov 03 18:14:45.898527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898399    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b5f5de-cb0f-440f-98d1-fead0c207e59" containerName="maas-api"
Nov 03 18:14:45.898527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898410    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy"
Nov 03 18:14:45.898527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898421    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" containerName="istio-proxy"
Nov 03 18:14:45.898527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.898430    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" containerName="istio-proxy"
Nov 03 18:14:45.903211 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.903189    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:45.959206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.959164    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-43bdc0-serving-cert\""
Nov 03 18:14:45.962436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.962408    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"]
Nov 03 18:14:45.984022 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.983986    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:45.984220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.984127    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpht\" (UniqueName: \"kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:45.984220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.984211    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:45.984332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:45.984264    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.085060 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.085020    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpht\" (UniqueName: \"kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.085284 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.085081    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.085284 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.085136    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.085284 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.085273    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.085457 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:46.085281    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-serving-cert: secret "maas-api-e2e-api-a-43bdc0-serving-cert" not found
Nov 03 18:14:46.085457 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:14:46.085355    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls podName:ee019581-201a-42e9-993d-dd262d39dfe7 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:46.58533324 +0000 UTC m=+2422.596071296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls") pod "maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" (UID: "ee019581-201a-42e9-993d-dd262d39dfe7") : secret "maas-api-e2e-api-a-43bdc0-serving-cert" not found
Nov 03 18:14:46.086243 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.086215    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.087883 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.087858    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.117908 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.117876    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04be7051-223e-4d83-9f57-fcefdc8470b2" path="/var/lib/kubelet/pods/04be7051-223e-4d83-9f57-fcefdc8470b2/volumes"
Nov 03 18:14:46.118375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.118362    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8effaaf-1861-4257-8784-3c0df65f2d30" path="/var/lib/kubelet/pods/a8effaaf-1861-4257-8784-3c0df65f2d30/volumes"
Nov 03 18:14:46.134669 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.134629    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:46.150118 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.150053    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpht\" (UniqueName: \"kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.339299 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.339236    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:46.496083 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.496001    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:46.591432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.591394    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.594179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.594153    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\") pod \"maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:46.817008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:46.816920    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:47.209443 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:14:47.209394    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee019581_201a_42e9_993d_dd262d39dfe7.slice/crio-06125d9d77380050191cfef9731ae712377973c2e49878e3e2303f1c9ab93ba4 WatchSource:0}: Error finding container 06125d9d77380050191cfef9731ae712377973c2e49878e3e2303f1c9ab93ba4: Status 404 returned error can't find the container with id 06125d9d77380050191cfef9731ae712377973c2e49878e3e2303f1c9ab93ba4
Nov 03 18:14:47.209612 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:47.209578    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"]
Nov 03 18:14:47.211191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:47.211168    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:14:47.266375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:47.266339    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" event={"ID":"ee019581-201a-42e9-993d-dd262d39dfe7","Type":"ContainerStarted","Data":"06125d9d77380050191cfef9731ae712377973c2e49878e3e2303f1c9ab93ba4"}
Nov 03 18:14:48.184986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:48.184956    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:14:48.185486 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:48.185022    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:14:48.277251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:48.277210    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" event={"ID":"ee019581-201a-42e9-993d-dd262d39dfe7","Type":"ContainerStarted","Data":"7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b"}
Nov 03 18:14:48.277458 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:48.277283    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:48.548670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:48.548614    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" podStartSLOduration=2.710773769 podStartE2EDuration="3.548596784s" totalImagesPullingTime="837.823015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:48.546007915 +0000 UTC m=+2424.556745988" watchObservedRunningTime="2026-11-03 18:14:48.548596784 +0000 UTC m=+2424.559334856"
Nov 03 18:14:49.563717 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:49.563678    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:49.721324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:49.721276    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:49.973524 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:49.973484    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:50.550354 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:50.550313    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:51.512392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:51.512351    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:54.290685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:54.290655    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:14:55.518308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:55.518263    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"]
Nov 03 18:14:55.677363 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:55.677324    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:56.769868 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:56.769803    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:57.024534 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:57.024402    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:59.251328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:14:59.251253    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:00.171641 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.171608    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"]
Nov 03 18:15:00.176052 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.176031    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.196807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.196773    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"]
Nov 03 18:15:00.200986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.200963    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.202863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.202838    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"]
Nov 03 18:15:00.209197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.209172    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.221862 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.221838    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"]
Nov 03 18:15:00.226071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.226052    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.250025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.249999    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:15:00.276662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.276067    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"]
Nov 03 18:15:00.282167 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.282135    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"]
Nov 03 18:15:00.285898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.285862    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-qjtjz\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.286212 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.286188    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.286834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.286717    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.286995 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.286959    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"]
Nov 03 18:15:00.287215 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.287000    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfb7\" (UniqueName: \"kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.287309 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.287276    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.287386 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.287315    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht46d\" (UniqueName: \"kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d\") pod \"maas-api-key-cleanup-29895495-qjtjz\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.287451 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.287407    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ft9d\" (UniqueName: \"kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.287451 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.287434    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9tg\" (UniqueName: \"kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.344522 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.344486    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"]
Nov 03 18:15:00.388383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388347    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ft9d\" (UniqueName: \"kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.388383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388381    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9tg\" (UniqueName: \"kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.388609 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388485    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-qjtjz\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.388609 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388511    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.388609 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388590    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.388795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388618    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfb7\" (UniqueName: \"kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.388795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388661    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.388795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.388684    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht46d\" (UniqueName: \"kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d\") pod \"maas-api-key-cleanup-29895495-qjtjz\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.389432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.389386    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.389432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.389402    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.389572 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.389481    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-qjtjz\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.389572 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.389488    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.452067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.451989    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfb7\" (UniqueName: \"kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.452067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.452019    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht46d\" (UniqueName: \"kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d\") pod \"maas-api-key-cleanup-29895495-qjtjz\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.452067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.452035    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ft9d\" (UniqueName: \"kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.452269 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.452086    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9tg\" (UniqueName: \"kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.491097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.491063    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:00.521567 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.521528    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:00.528064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.527887    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:00.545277 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.545233    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:00.808603 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.808510    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:00.884760 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:15:00.884691    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711fa367_1899_4a42_9f96_cf28dbd0998e.slice/crio-2e9522c465bd5da9af330ecd1d4cf84f2081aebb747451ea93dc0368889746c9 WatchSource:0}: Error finding container 2e9522c465bd5da9af330ecd1d4cf84f2081aebb747451ea93dc0368889746c9: Status 404 returned error can't find the container with id 2e9522c465bd5da9af330ecd1d4cf84f2081aebb747451ea93dc0368889746c9
Nov 03 18:15:00.885569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.885539    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"]
Nov 03 18:15:00.966705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.966675    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"]
Nov 03 18:15:00.967668 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:15:00.967638    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3805976d_1f09_4603_ab4d_07154e278069.slice/crio-25588a49ca929bbd6af8f0dfbfa3b3dde1cc1944fb6d7d112c4a7644f953679a WatchSource:0}: Error finding container 25588a49ca929bbd6af8f0dfbfa3b3dde1cc1944fb6d7d112c4a7644f953679a: Status 404 returned error can't find the container with id 25588a49ca929bbd6af8f0dfbfa3b3dde1cc1944fb6d7d112c4a7644f953679a
Nov 03 18:15:00.995588 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:00.995556    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"]
Nov 03 18:15:01.000857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:01.000825    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"]
Nov 03 18:15:01.397984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:01.397190    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerStarted","Data":"25588a49ca929bbd6af8f0dfbfa3b3dde1cc1944fb6d7d112c4a7644f953679a"}
Nov 03 18:15:01.412781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:01.408562    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerStarted","Data":"2e9522c465bd5da9af330ecd1d4cf84f2081aebb747451ea93dc0368889746c9"}
Nov 03 18:15:01.412781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:01.410527    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerStarted","Data":"a9b173e385f3b8655dd3a136cdef857e52c38144462a939693ea576e0112de30"}
Nov 03 18:15:01.427134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:01.427096    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerStarted","Data":"5a985d310c552410501d037723f41ce6d0aae1553332e11721eade1f16d9573c"}
Nov 03 18:15:02.116912 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.116879    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:02.329520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.329413    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:02.436939 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.436907    5838 generic.go:356] "Generic (PLEG): container finished" podID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerID="ce25868ee83f3b64925a3d29e50b08ec2193f6e0d40d64596f7cc7b9500f6c3c" exitCode=60
Nov 03 18:15:02.437385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.436948    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerDied","Data":"ce25868ee83f3b64925a3d29e50b08ec2193f6e0d40d64596f7cc7b9500f6c3c"}
Nov 03 18:15:02.437385 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.437305    5838 scope.go:122] "RemoveContainer" containerID="ce25868ee83f3b64925a3d29e50b08ec2193f6e0d40d64596f7cc7b9500f6c3c"
Nov 03 18:15:02.438967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.438924    5838 generic.go:356] "Generic (PLEG): container finished" podID="3805976d-1f09-4603-ab4d-07154e278069" containerID="ed1e03232a4132b5fa93cb5a7e919b2ec9a2fe511fc8d27606447eb5431ff59b" exitCode=60
Nov 03 18:15:02.439162 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.438997    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerDied","Data":"ed1e03232a4132b5fa93cb5a7e919b2ec9a2fe511fc8d27606447eb5431ff59b"}
Nov 03 18:15:02.439272 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.439253    5838 scope.go:122] "RemoveContainer" containerID="ed1e03232a4132b5fa93cb5a7e919b2ec9a2fe511fc8d27606447eb5431ff59b"
Nov 03 18:15:02.440809 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.440788    5838 generic.go:356] "Generic (PLEG): container finished" podID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerID="cc60444e3018c43f5feb7c1a01ad36177aa10ddbb0329b5f37eceaa39986b97c" exitCode=60
Nov 03 18:15:02.440898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.440827    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerDied","Data":"cc60444e3018c43f5feb7c1a01ad36177aa10ddbb0329b5f37eceaa39986b97c"}
Nov 03 18:15:02.441106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.441075    5838 scope.go:122] "RemoveContainer" containerID="cc60444e3018c43f5feb7c1a01ad36177aa10ddbb0329b5f37eceaa39986b97c"
Nov 03 18:15:02.442923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.442900    5838 generic.go:356] "Generic (PLEG): container finished" podID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerID="83fa62b7c7b8321f147d5dcb98a42f755cfbb2f82c0955379c7ac4e9d25532db" exitCode=60
Nov 03 18:15:02.443003 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.442946    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerDied","Data":"83fa62b7c7b8321f147d5dcb98a42f755cfbb2f82c0955379c7ac4e9d25532db"}
Nov 03 18:15:02.443256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:02.443240    5838 scope.go:122] "RemoveContainer" containerID="83fa62b7c7b8321f147d5dcb98a42f755cfbb2f82c0955379c7ac4e9d25532db"
Nov 03 18:15:03.454511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.454473    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerStarted","Data":"1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8"}
Nov 03 18:15:03.456763 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.456712    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerStarted","Data":"78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444"}
Nov 03 18:15:03.458924 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.458895    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerStarted","Data":"6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167"}
Nov 03 18:15:03.461059 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.461039    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerStarted","Data":"0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d"}
Nov 03 18:15:03.576194 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.576128    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" podStartSLOduration=3.5761079159999998 podStartE2EDuration="3.576107916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.572282822 +0000 UTC m=+2439.583020915" watchObservedRunningTime="2026-11-03 18:15:03.576107916 +0000 UTC m=+2439.586845991"
Nov 03 18:15:03.692963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.692909    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" podStartSLOduration=3.692890384 podStartE2EDuration="3.692890384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.690643452 +0000 UTC m=+2439.701381551" watchObservedRunningTime="2026-11-03 18:15:03.692890384 +0000 UTC m=+2439.703628456"
Nov 03 18:15:03.812545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.812470    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" podStartSLOduration=3.812453401 podStartE2EDuration="3.812453401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.811137577 +0000 UTC m=+2439.821875662" watchObservedRunningTime="2026-11-03 18:15:03.812453401 +0000 UTC m=+2439.823191472"
Nov 03 18:15:03.930509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:03.930442    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" podStartSLOduration=3.930426912 podStartE2EDuration="3.930426912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.928630811 +0000 UTC m=+2439.939368907" watchObservedRunningTime="2026-11-03 18:15:03.930426912 +0000 UTC m=+2439.941165003"
Nov 03 18:15:04.470756 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.470697    5838 generic.go:356] "Generic (PLEG): container finished" podID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerID="0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d" exitCode=60
Nov 03 18:15:04.471254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.470775    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerDied","Data":"0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d"}
Nov 03 18:15:04.471254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.470867    5838 scope.go:122] "RemoveContainer" containerID="ce25868ee83f3b64925a3d29e50b08ec2193f6e0d40d64596f7cc7b9500f6c3c"
Nov 03 18:15:04.471254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.471203    5838 scope.go:122] "RemoveContainer" containerID="0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d"
Nov 03 18:15:04.471521 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:04.471498    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794_odh-ai-gateway-infra(3e32f1ed-b44e-464a-8144-8495dbcb15b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7"
Nov 03 18:15:04.473351 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.473326    5838 generic.go:356] "Generic (PLEG): container finished" podID="3805976d-1f09-4603-ab4d-07154e278069" containerID="1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8" exitCode=60
Nov 03 18:15:04.473460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.473383    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerDied","Data":"1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8"}
Nov 03 18:15:04.473706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.473690    5838 scope.go:122] "RemoveContainer" containerID="1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8"
Nov 03 18:15:04.474045 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:04.474027    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-qjtjz_odh-ai-gateway-infra(3805976d-1f09-4603-ab4d-07154e278069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" podUID="3805976d-1f09-4603-ab4d-07154e278069"
Nov 03 18:15:04.475523 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.475498    5838 generic.go:356] "Generic (PLEG): container finished" podID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerID="78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444" exitCode=60
Nov 03 18:15:04.475632 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.475567    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerDied","Data":"78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444"}
Nov 03 18:15:04.476045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.476026    5838 scope.go:122] "RemoveContainer" containerID="78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444"
Nov 03 18:15:04.476400 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:04.476376    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj_odh-ai-gateway-infra(711fa367-1899-4a42-9f96-cf28dbd0998e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e"
Nov 03 18:15:04.477545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.477525    5838 generic.go:356] "Generic (PLEG): container finished" podID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerID="6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167" exitCode=60
Nov 03 18:15:04.477643 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.477567    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerDied","Data":"6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167"}
Nov 03 18:15:04.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.477860    5838 scope.go:122] "RemoveContainer" containerID="6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167"
Nov 03 18:15:04.478142 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:04.478120    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn_odh-ai-gateway-infra(0842e288-6a80-4c7d-b7c7-0fe1a762d15c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c"
Nov 03 18:15:04.484882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.484863    5838 scope.go:122] "RemoveContainer" containerID="ed1e03232a4132b5fa93cb5a7e919b2ec9a2fe511fc8d27606447eb5431ff59b"
Nov 03 18:15:04.501114 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.501091    5838 scope.go:122] "RemoveContainer" containerID="cc60444e3018c43f5feb7c1a01ad36177aa10ddbb0329b5f37eceaa39986b97c"
Nov 03 18:15:04.514611 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:04.514587    5838 scope.go:122] "RemoveContainer" containerID="83fa62b7c7b8321f147d5dcb98a42f755cfbb2f82c0955379c7ac4e9d25532db"
Nov 03 18:15:05.081500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.081461    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:05.236147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.236115    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:05.488996 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.488963    5838 scope.go:122] "RemoveContainer" containerID="1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8"
Nov 03 18:15:05.489471 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:05.489308    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-qjtjz_odh-ai-gateway-infra(3805976d-1f09-4603-ab4d-07154e278069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" podUID="3805976d-1f09-4603-ab4d-07154e278069"
Nov 03 18:15:05.490766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.490743    5838 scope.go:122] "RemoveContainer" containerID="78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444"
Nov 03 18:15:05.491057 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:05.491027    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj_odh-ai-gateway-infra(711fa367-1899-4a42-9f96-cf28dbd0998e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e"
Nov 03 18:15:05.491951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.491925    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:05.492511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.492493    5838 scope.go:122] "RemoveContainer" containerID="6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167"
Nov 03 18:15:05.492725 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:05.492707    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn_odh-ai-gateway-infra(0842e288-6a80-4c7d-b7c7-0fe1a762d15c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c"
Nov 03 18:15:05.494008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:05.493994    5838 scope.go:122] "RemoveContainer" containerID="0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d"
Nov 03 18:15:05.494195 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:05.494179    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794_odh-ai-gateway-infra(3e32f1ed-b44e-464a-8144-8495dbcb15b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7"
Nov 03 18:15:06.364675 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:06.364641    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:07.040890 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:07.040691    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:07.385287 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:07.385173    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:08.165936 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:08.165895    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:08.657812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:08.657725    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:08.813434 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:08.813391    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:09.031671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:09.031587    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"]
Nov 03 18:15:09.031914 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:09.031853    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" containerID="cri-o://8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4" gracePeriod=30
Nov 03 18:15:09.042254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:09.042222    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:09.047721 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:09.047694    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"]
Nov 03 18:15:09.047940 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:09.047918    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" containerID="cri-o://ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec" gracePeriod=30
Nov 03 18:15:10.345164 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:10.345121    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:15:10.540449 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.540411    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main" containerID="cri-o://4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb" gracePeriod=45
Nov 03 18:15:10.817812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.817411    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 18:15:10.953403 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953312    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm\") pod \"ee1d2385-7689-4ff5-8f33-8c043448c25c\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") "
Nov 03 18:15:10.953403 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953368    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir\") pod \"ee1d2385-7689-4ff5-8f33-8c043448c25c\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") "
Nov 03 18:15:10.953623 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953428    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache\") pod \"ee1d2385-7689-4ff5-8f33-8c043448c25c\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") "
Nov 03 18:15:10.953711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953685    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache" pod "ee1d2385-7689-4ff5-8f33-8c043448c25c" (UID: "ee1d2385-7689-4ff5-8f33-8c043448c25c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.953812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953704    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir" pod "ee1d2385-7689-4ff5-8f33-8c043448c25c" (UID: "ee1d2385-7689-4ff5-8f33-8c043448c25c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.953812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953801    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home\") pod \"ee1d2385-7689-4ff5-8f33-8c043448c25c\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") "
Nov 03 18:15:10.953923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953858    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq\" (UniqueName: \"kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq\") pod \"ee1d2385-7689-4ff5-8f33-8c043448c25c\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") "
Nov 03 18:15:10.954033 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.953980    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs\") pod \"ee1d2385-7689-4ff5-8f33-8c043448c25c\" (UID: \"ee1d2385-7689-4ff5-8f33-8c043448c25c\") "
Nov 03 18:15:10.954276 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.954055    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home" pod "ee1d2385-7689-4ff5-8f33-8c043448c25c" (UID: "ee1d2385-7689-4ff5-8f33-8c043448c25c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.954671 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.954582    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.954804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.954663    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.954804 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.954798    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.956035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.956007    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm" pod "ee1d2385-7689-4ff5-8f33-8c043448c25c" (UID: "ee1d2385-7689-4ff5-8f33-8c043448c25c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.956260 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.956241    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs" pod "ee1d2385-7689-4ff5-8f33-8c043448c25c" (UID: "ee1d2385-7689-4ff5-8f33-8c043448c25c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:10.956574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:10.956544    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq" pod "ee1d2385-7689-4ff5-8f33-8c043448c25c" (UID: "ee1d2385-7689-4ff5-8f33-8c043448c25c"). InnerVolumeSpecName "kube-api-access-zdzcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:11.056456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.056422    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzcq\" (UniqueName: \"kubernetes.io/projected/ee1d2385-7689-4ff5-8f33-8c043448c25c-kube-api-access-zdzcq\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.056456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.056453    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1d2385-7689-4ff5-8f33-8c043448c25c-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.056456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.056462    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1d2385-7689-4ff5-8f33-8c043448c25c-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.547138 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.547096    5838 generic.go:356] "Generic (PLEG): container finished" podID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerID="4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb" exitCode=0
Nov 03 18:15:11.547138 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.547135    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"
Nov 03 18:15:11.547725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.547180    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" event={"ID":"ee1d2385-7689-4ff5-8f33-8c043448c25c","Type":"ContainerDied","Data":"4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb"}
Nov 03 18:15:11.547725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.547226    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d" event={"ID":"ee1d2385-7689-4ff5-8f33-8c043448c25c","Type":"ContainerDied","Data":"41cf8e3b16e98abf8b32069706c2bc58ca360c92d9aa9405cd531608b49d468a"}
Nov 03 18:15:11.547725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.547252    5838 scope.go:122] "RemoveContainer" containerID="4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb"
Nov 03 18:15:11.558573 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.558551    5838 scope.go:122] "RemoveContainer" containerID="4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb"
Nov 03 18:15:11.558999 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:11.558961    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb\": container with ID starting with 4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb not found: ID does not exist" containerID="4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb"
Nov 03 18:15:11.559125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.558995    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb"} err="failed to get container status \"4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb\": rpc error: code = NotFound desc = could not find container \"4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb\": container with ID starting with 4b4e00329d724e1ca77e56b9ebe0f7aa018b222e6840a0a79df4bc032736e8fb not found: ID does not exist"
Nov 03 18:15:11.717897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.717860    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"]
Nov 03 18:15:11.767809 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.767773    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-w5-548ca1-kserve-5cdf8f8cdb-krf5d"]
Nov 03 18:15:11.805784 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.805671    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:15:11.805944 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.805783    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:11.824066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.824033    5838 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:15:11.824237 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:11.824087    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:12.075563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:12.075459    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:12.119248 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:12.119214    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" path="/var/lib/kubelet/pods/ee1d2385-7689-4ff5-8f33-8c043448c25c/volumes"
Nov 03 18:15:12.248039 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:12.248007    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:12.411494 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:12.411444    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:13.497047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:13.497004    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"]
Nov 03 18:15:13.761388 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:13.761276    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:13.916834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:13.916796    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:14.322780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.322700    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:15:14.391238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.391210    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:15:14.503837 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.503806    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.503876    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.503926    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.503967    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504038    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504110    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504140    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504166    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504194    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504247    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504289    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504330    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.504335 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504329    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504391    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504463    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504498    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504557    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504615    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket\") pod \"d40b6751-bcec-4589-91b5-0b058c080980\" (UID: \"d40b6751-bcec-4589-91b5-0b058c080980\") "
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504643    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.505051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.504654    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket\") pod \"e3b48127-ba3d-4760-9c93-a497c27e9321\" (UID: \"e3b48127-ba3d-4760-9c93-a497c27e9321\") "
Nov 03 18:15:14.505386 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.505244    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40b6751-bcec-4589-91b5-0b058c080980-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.505386 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.505267    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.505494 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.505376    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:14.505547 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.505525    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.505905 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.505718    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.507143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.506572    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.507143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.506634    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.507143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.506676    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.507143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.506885    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.507143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.506922    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.507606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.507578    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:14.509154 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.509127    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:14.509632 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.509343    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "kube-api-access-r4kl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.510010 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.509873    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.510104 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.510018    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "kube-api-access-jtkbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.510236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.510213    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy" pod "d40b6751-bcec-4589-91b5-0b058c080980" (UID: "d40b6751-bcec-4589-91b5-0b058c080980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.510580 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.510565    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.511132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.511115    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token" pod "e3b48127-ba3d-4760-9c93-a497c27e9321" (UID: "e3b48127-ba3d-4760-9c93-a497c27e9321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.590149 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.590067    5838 generic.go:356] "Generic (PLEG): container finished" podID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerID="8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4" exitCode=0
Nov 03 18:15:14.590326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.590196    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" event={"ID":"e3b48127-ba3d-4760-9c93-a497c27e9321","Type":"ContainerDied","Data":"8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4"}
Nov 03 18:15:14.590326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.590249    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw" event={"ID":"e3b48127-ba3d-4760-9c93-a497c27e9321","Type":"ContainerDied","Data":"83d4596f890953a9e130acbeadd44653ed0ebe2cb1e50a73a44683f282af2123"}
Nov 03 18:15:14.590326 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.590276    5838 scope.go:122] "RemoveContainer" containerID="8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4"
Nov 03 18:15:14.590491 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.590429    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"
Nov 03 18:15:14.594841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.594807    5838 generic.go:356] "Generic (PLEG): container finished" podID="d40b6751-bcec-4589-91b5-0b058c080980" containerID="ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec" exitCode=0
Nov 03 18:15:14.595044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.594856    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"
Nov 03 18:15:14.595193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.594855    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" event={"ID":"d40b6751-bcec-4589-91b5-0b058c080980","Type":"ContainerDied","Data":"ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec"}
Nov 03 18:15:14.595193 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.595173    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85" event={"ID":"d40b6751-bcec-4589-91b5-0b058c080980","Type":"ContainerDied","Data":"53363035592b03aace8d4dffcdebc722fe1745d603d6d6e5e52afe65a76bd896"}
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607363    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607400    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607418    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607435    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607451    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kl2\" (UniqueName: \"kubernetes.io/projected/d40b6751-bcec-4589-91b5-0b058c080980-kube-api-access-r4kl2\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607467    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607481    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607497    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607512    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40b6751-bcec-4589-91b5-0b058c080980-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607526    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607540    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40b6751-bcec-4589-91b5-0b058c080980-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607556    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607571    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkbg\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-kube-api-access-jtkbg\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607585    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b48127-ba3d-4760-9c93-a497c27e9321-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607599    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b48127-ba3d-4760-9c93-a497c27e9321-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.607636 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.607613    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b48127-ba3d-4760-9c93-a497c27e9321-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.610309 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.610275    5838 scope.go:122] "RemoveContainer" containerID="8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4"
Nov 03 18:15:14.610724 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:14.610691    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4\": container with ID starting with 8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4 not found: ID does not exist" containerID="8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4"
Nov 03 18:15:14.610865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.610775    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4"} err="failed to get container status \"8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4\": rpc error: code = NotFound desc = could not find container \"8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4\": container with ID starting with 8fbeee484333f602bd63086af83c945293078b51ddf518ed1d33f18d938163a4 not found: ID does not exist"
Nov 03 18:15:14.610865 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.610802    5838 scope.go:122] "RemoveContainer" containerID="ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec"
Nov 03 18:15:14.624333 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.624306    5838 scope.go:122] "RemoveContainer" containerID="ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec"
Nov 03 18:15:14.624692 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:14.624665    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec\": container with ID starting with ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec not found: ID does not exist" containerID="ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec"
Nov 03 18:15:14.624812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.624704    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec"} err="failed to get container status \"ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec\": rpc error: code = NotFound desc = could not find container \"ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec\": container with ID starting with ac3de1a9978d67afd6420e05ed357f710bce39253b9698a1a0d46285628795ec not found: ID does not exist"
Nov 03 18:15:14.764192 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.764157    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"]
Nov 03 18:15:14.771998 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.771966    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-pzg85"]
Nov 03 18:15:14.885481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.885392    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"]
Nov 03 18:15:14.894948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:14.894916    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-4vrlw"]
Nov 03 18:15:15.108924 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:15.108884    5838 scope.go:122] "RemoveContainer" containerID="78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444"
Nov 03 18:15:15.109082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:15.108957    5838 scope.go:122] "RemoveContainer" containerID="6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167"
Nov 03 18:15:15.607641 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:15.607606    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerStarted","Data":"4be75e3cb16e4d280213cd19ab1b7a50049b1aa148bfc747dbbeea56743f8896"}
Nov 03 18:15:15.609452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:15.609427    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerStarted","Data":"a1f2303170a16411e1e1f90c1a59a3fa92f0d537fc5ba1df60b79ec2e8faf27d"}
Nov 03 18:15:16.109245 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.109204    5838 scope.go:122] "RemoveContainer" containerID="0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d"
Nov 03 18:15:16.117708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.117678    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40b6751-bcec-4589-91b5-0b058c080980" path="/var/lib/kubelet/pods/d40b6751-bcec-4589-91b5-0b058c080980/volumes"
Nov 03 18:15:16.118171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.118157    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" path="/var/lib/kubelet/pods/e3b48127-ba3d-4760-9c93-a497c27e9321/volumes"
Nov 03 18:15:16.620383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.620299    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerStarted","Data":"457d4d541eac82128ca2aa4513779219e4cc2702348e86d958d8cb2645f0b09b"}
Nov 03 18:15:16.622465 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.622438    5838 generic.go:356] "Generic (PLEG): container finished" podID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerID="4be75e3cb16e4d280213cd19ab1b7a50049b1aa148bfc747dbbeea56743f8896" exitCode=60
Nov 03 18:15:16.622654 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.622518    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerDied","Data":"4be75e3cb16e4d280213cd19ab1b7a50049b1aa148bfc747dbbeea56743f8896"}
Nov 03 18:15:16.622654 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.622571    5838 scope.go:122] "RemoveContainer" containerID="78409515faaef3b1c21a5d230fdd8f99e1669f792fe036a2f0850543cf560444"
Nov 03 18:15:16.622956 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.622923    5838 scope.go:122] "RemoveContainer" containerID="4be75e3cb16e4d280213cd19ab1b7a50049b1aa148bfc747dbbeea56743f8896"
Nov 03 18:15:16.623289 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:16.623256    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj_odh-ai-gateway-infra(711fa367-1899-4a42-9f96-cf28dbd0998e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e"
Nov 03 18:15:16.624692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.624672    5838 generic.go:356] "Generic (PLEG): container finished" podID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerID="a1f2303170a16411e1e1f90c1a59a3fa92f0d537fc5ba1df60b79ec2e8faf27d" exitCode=60
Nov 03 18:15:16.624807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.624766    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerDied","Data":"a1f2303170a16411e1e1f90c1a59a3fa92f0d537fc5ba1df60b79ec2e8faf27d"}
Nov 03 18:15:16.625068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.625055    5838 scope.go:122] "RemoveContainer" containerID="a1f2303170a16411e1e1f90c1a59a3fa92f0d537fc5ba1df60b79ec2e8faf27d"
Nov 03 18:15:16.625279 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:16.625263    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn_odh-ai-gateway-infra(0842e288-6a80-4c7d-b7c7-0fe1a762d15c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c"
Nov 03 18:15:16.636008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.635989    5838 scope.go:122] "RemoveContainer" containerID="6a2d7f20cb4568a807464a04fe0b4f8669f6b38be6ab322d78b2d837fa81e167"
Nov 03 18:15:16.739286 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.739254    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"]
Nov 03 18:15:16.867147 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:16.867108    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"]
Nov 03 18:15:17.109273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.109249    5838 scope.go:122] "RemoveContainer" containerID="1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8"
Nov 03 18:15:17.325664 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.325628    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:17.502530 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.502497    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:17.650241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.650177    5838 generic.go:356] "Generic (PLEG): container finished" podID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerID="457d4d541eac82128ca2aa4513779219e4cc2702348e86d958d8cb2645f0b09b" exitCode=60
Nov 03 18:15:17.650708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.650437    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerDied","Data":"457d4d541eac82128ca2aa4513779219e4cc2702348e86d958d8cb2645f0b09b"}
Nov 03 18:15:17.650708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.650498    5838 scope.go:122] "RemoveContainer" containerID="0321ee5caec357a5f01655e6cb94e77a01624992853f9441c15670bbebb9568d"
Nov 03 18:15:17.651050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.650984    5838 scope.go:122] "RemoveContainer" containerID="457d4d541eac82128ca2aa4513779219e4cc2702348e86d958d8cb2645f0b09b"
Nov 03 18:15:17.651316 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:17.651288    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794_odh-ai-gateway-infra(3e32f1ed-b44e-464a-8144-8495dbcb15b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7"
Nov 03 18:15:17.655040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.654399    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerStarted","Data":"57024dffbc2517a0913e66ef0d249b38ffed1a6167e625799a12cd419b0b3e89"}
Nov 03 18:15:17.736313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.736274    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"]
Nov 03 18:15:17.737271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737252    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main"
Nov 03 18:15:17.737352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737272    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main"
Nov 03 18:15:17.737352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737293    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy"
Nov 03 18:15:17.737352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737301    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy"
Nov 03 18:15:17.737352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737336    5838 cpu_manager.go:414] "containerMap: removing container" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy"
Nov 03 18:15:17.737352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737344    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy"
Nov 03 18:15:17.737538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737465    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1d2385-7689-4ff5-8f33-8c043448c25c" containerName="main"
Nov 03 18:15:17.737538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737476    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b48127-ba3d-4760-9c93-a497c27e9321" containerName="istio-proxy"
Nov 03 18:15:17.737538 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.737494    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40b6751-bcec-4589-91b5-0b058c080980" containerName="istio-proxy"
Nov 03 18:15:17.742446 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.742404    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.752401 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.749474    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.752401 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.749591    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwhm\" (UniqueName: \"kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm\") pod \"payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.754812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.754772    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"]
Nov 03 18:15:17.799141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.799012    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-44769b\""
Nov 03 18:15:17.799141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.799071    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-44769b-dockercfg-h8jkt\""
Nov 03 18:15:17.802932 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.800698    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"]
Nov 03 18:15:17.879806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.875890    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.879806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.876009    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwhm\" (UniqueName: \"kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm\") pod \"payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.879806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.877377    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.904520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.904475    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"]
Nov 03 18:15:17.928312 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.928166    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:17.929114 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.929082    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:17.963310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.944049    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwhm\" (UniqueName: \"kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm\") pod \"payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:17.963310 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:17.946711    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"]
Nov 03 18:15:18.005861 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.005832    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:18.077606 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.077572    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:18.077889 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.077866    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca\") pod \"711fa367-1899-4a42-9f96-cf28dbd0998e\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") "
Nov 03 18:15:18.077941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.077900    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d\" (UniqueName: \"kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d\") pod \"711fa367-1899-4a42-9f96-cf28dbd0998e\" (UID: \"711fa367-1899-4a42-9f96-cf28dbd0998e\") "
Nov 03 18:15:18.078274 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.078251    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca" pod "711fa367-1899-4a42-9f96-cf28dbd0998e" (UID: "711fa367-1899-4a42-9f96-cf28dbd0998e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:18.078399 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.078383    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-44769b-69576fcb85-nxqct\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") " pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.078498 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.078486    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ts8r\" (UniqueName: \"kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r\") pod \"payload-processing-e2e-api-b-44769b-69576fcb85-nxqct\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") " pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.078662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.078651    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711fa367-1899-4a42-9f96-cf28dbd0998e-maas-api-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.080296 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.080272    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d" pod "711fa367-1899-4a42-9f96-cf28dbd0998e" (UID: "711fa367-1899-4a42-9f96-cf28dbd0998e"). InnerVolumeSpecName "kube-api-access-6ft9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.186152    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg\" (UniqueName: \"kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg\") pod \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") "
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.186397    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca\") pod \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\" (UID: \"0842e288-6a80-4c7d-b7c7-0fe1a762d15c\") "
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.186622    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-44769b-69576fcb85-nxqct\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") " pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.186719    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ts8r\" (UniqueName: \"kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r\") pod \"payload-processing-e2e-api-b-44769b-69576fcb85-nxqct\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") " pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.187040    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ft9d\" (UniqueName: \"kubernetes.io/projected/711fa367-1899-4a42-9f96-cf28dbd0998e-kube-api-access-6ft9d\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.187576    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca" pod "0842e288-6a80-4c7d-b7c7-0fe1a762d15c" (UID: "0842e288-6a80-4c7d-b7c7-0fe1a762d15c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:18.188420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.188292    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-44769b-69576fcb85-nxqct\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") " pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.192224 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.192197    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg" pod "0842e288-6a80-4c7d-b7c7-0fe1a762d15c" (UID: "0842e288-6a80-4c7d-b7c7-0fe1a762d15c"). InnerVolumeSpecName "kube-api-access-cc9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:18.250788 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.250714    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ts8r\" (UniqueName: \"kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r\") pod \"payload-processing-e2e-api-b-44769b-69576fcb85-nxqct\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") " pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.288214 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.288180    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-maas-api-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.288214 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.288210    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9tg\" (UniqueName: \"kubernetes.io/projected/0842e288-6a80-4c7d-b7c7-0fe1a762d15c-kube-api-access-cc9tg\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.296611 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.296570    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:18.464983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.464948    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"]
Nov 03 18:15:18.466027 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:15:18.466002    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645817b9_1596_4cf5_bf55_e33d0deb6658.slice/crio-49ddeb9ffb3b38fee58d2cd77440ea26217dd0e49da467360a580a5891f9c78f WatchSource:0}: Error finding container 49ddeb9ffb3b38fee58d2cd77440ea26217dd0e49da467360a580a5891f9c78f: Status 404 returned error can't find the container with id 49ddeb9ffb3b38fee58d2cd77440ea26217dd0e49da467360a580a5891f9c78f
Nov 03 18:15:18.503867 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.503815    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" podUID="7c40819a-98af-412e-8542-ccf40133be24" containerName="payload-pre-processing" containerID="cri-o://d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3" gracePeriod=25
Nov 03 18:15:18.668483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.668450    5838 generic.go:356] "Generic (PLEG): container finished" podID="3805976d-1f09-4603-ab4d-07154e278069" containerID="57024dffbc2517a0913e66ef0d249b38ffed1a6167e625799a12cd419b0b3e89" exitCode=60
Nov 03 18:15:18.668947 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.668519    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerDied","Data":"57024dffbc2517a0913e66ef0d249b38ffed1a6167e625799a12cd419b0b3e89"}
Nov 03 18:15:18.668947 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.668631    5838 scope.go:122] "RemoveContainer" containerID="1744c6f586984ff72ed7b979eac32527924204a00c87ca9b9a1c436b08fa19f8"
Nov 03 18:15:18.669071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.669045    5838 scope.go:122] "RemoveContainer" containerID="57024dffbc2517a0913e66ef0d249b38ffed1a6167e625799a12cd419b0b3e89"
Nov 03 18:15:18.669349 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:18.669326    5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-qjtjz_odh-ai-gateway-infra(3805976d-1f09-4603-ab4d-07154e278069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" podUID="3805976d-1f09-4603-ab4d-07154e278069"
Nov 03 18:15:18.670582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.670531    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj" event={"ID":"711fa367-1899-4a42-9f96-cf28dbd0998e","Type":"ContainerDied","Data":"2e9522c465bd5da9af330ecd1d4cf84f2081aebb747451ea93dc0368889746c9"}
Nov 03 18:15:18.670582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.670547    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"
Nov 03 18:15:18.672425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.672384    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn" event={"ID":"0842e288-6a80-4c7d-b7c7-0fe1a762d15c","Type":"ContainerDied","Data":"a9b173e385f3b8655dd3a136cdef857e52c38144462a939693ea576e0112de30"}
Nov 03 18:15:18.672425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.672413    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"
Nov 03 18:15:18.675115 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.675084    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt" event={"ID":"645817b9-1596-4cf5-bf55-e33d0deb6658","Type":"ContainerStarted","Data":"e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54"}
Nov 03 18:15:18.675236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.675140    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt" event={"ID":"645817b9-1596-4cf5-bf55-e33d0deb6658","Type":"ContainerStarted","Data":"49ddeb9ffb3b38fee58d2cd77440ea26217dd0e49da467360a580a5891f9c78f"}
Nov 03 18:15:18.675236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.675194    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:18.683743 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.683632    5838 scope.go:122] "RemoveContainer" containerID="4be75e3cb16e4d280213cd19ab1b7a50049b1aa148bfc747dbbeea56743f8896"
Nov 03 18:15:18.695178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.695148    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"]
Nov 03 18:15:18.703450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.703416    5838 scope.go:122] "RemoveContainer" containerID="a1f2303170a16411e1e1f90c1a59a3fa92f0d537fc5ba1df60b79ec2e8faf27d"
Nov 03 18:15:18.824612 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.824587    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:18.901808 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.901775    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7\" (UniqueName: \"kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7\") pod \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") "
Nov 03 18:15:18.902062 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.902041    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca\") pod \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\" (UID: \"3e32f1ed-b44e-464a-8144-8495dbcb15b7\") "
Nov 03 18:15:18.902404 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.902378    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca" pod "3e32f1ed-b44e-464a-8144-8495dbcb15b7" (UID: "3e32f1ed-b44e-464a-8144-8495dbcb15b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:18.904104 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.904080    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7" pod "3e32f1ed-b44e-464a-8144-8495dbcb15b7" (UID: "3e32f1ed-b44e-464a-8144-8495dbcb15b7"). InnerVolumeSpecName "kube-api-access-zkfb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:18.910705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:18.910655    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt" podStartSLOduration=1.910638348 podStartE2EDuration="1.910638348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:18.907933483 +0000 UTC m=+2454.918671588" watchObservedRunningTime="2026-11-03 18:15:18.910638348 +0000 UTC m=+2454.921376423"
Nov 03 18:15:19.003579 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.003489    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e32f1ed-b44e-464a-8144-8495dbcb15b7-maas-api-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.003579 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.003521    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfb7\" (UniqueName: \"kubernetes.io/projected/3e32f1ed-b44e-464a-8144-8495dbcb15b7-kube-api-access-zkfb7\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.034288 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.034254    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"]
Nov 03 18:15:19.040601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.040564    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895495-qdvjj"]
Nov 03 18:15:19.149056 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.149022    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"]
Nov 03 18:15:19.155882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.155844    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895495-xvwbn"]
Nov 03 18:15:19.263878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.263780    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"]
Nov 03 18:15:19.595154 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.595046    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:19.687652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.687620    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"
Nov 03 18:15:19.688132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.687669    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794" event={"ID":"3e32f1ed-b44e-464a-8144-8495dbcb15b7","Type":"ContainerDied","Data":"5a985d310c552410501d037723f41ce6d0aae1553332e11721eade1f16d9573c"}
Nov 03 18:15:19.688132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.687767    5838 scope.go:122] "RemoveContainer" containerID="457d4d541eac82128ca2aa4513779219e4cc2702348e86d958d8cb2645f0b09b"
Nov 03 18:15:19.689827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.689798    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct" event={"ID":"576d89a7-d224-40ec-9206-7cf0eb45fa93","Type":"ContainerStarted","Data":"a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221"}
Nov 03 18:15:19.689958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.689850    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct" event={"ID":"576d89a7-d224-40ec-9206-7cf0eb45fa93","Type":"ContainerStarted","Data":"4932d38b0d74c585ab247f5d5cc3c0111c89bf77fa8b5b13c8eec38cfe8fece5"}
Nov 03 18:15:19.689958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.689896    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:19.853251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.853181    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:19.914570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.914533    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d\" (UniqueName: \"kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d\") pod \"3805976d-1f09-4603-ab4d-07154e278069\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") "
Nov 03 18:15:19.914789 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.914726    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca\") pod \"3805976d-1f09-4603-ab4d-07154e278069\" (UID: \"3805976d-1f09-4603-ab4d-07154e278069\") "
Nov 03 18:15:19.915112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.915090    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca" pod "3805976d-1f09-4603-ab4d-07154e278069" (UID: "3805976d-1f09-4603-ab4d-07154e278069"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:19.917330 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.917301    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d" pod "3805976d-1f09-4603-ab4d-07154e278069" (UID: "3805976d-1f09-4603-ab4d-07154e278069"). InnerVolumeSpecName "kube-api-access-ht46d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:19.980273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.980232    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"]
Nov 03 18:15:19.987786 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:19.987749    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895495-gw794"]
Nov 03 18:15:20.016577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.016544    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3805976d-1f09-4603-ab4d-07154e278069-maas-api-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.016577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.016577    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht46d\" (UniqueName: \"kubernetes.io/projected/3805976d-1f09-4603-ab4d-07154e278069-kube-api-access-ht46d\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.104045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.103921    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct" podStartSLOduration=3.103904701 podStartE2EDuration="3.103904701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:20.102032044 +0000 UTC m=+2456.112770116" watchObservedRunningTime="2026-11-03 18:15:20.103904701 +0000 UTC m=+2456.114642750"
Nov 03 18:15:20.118664 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.118623    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" path="/var/lib/kubelet/pods/0842e288-6a80-4c7d-b7c7-0fe1a762d15c/volumes"
Nov 03 18:15:20.119206 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.119188    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" path="/var/lib/kubelet/pods/3e32f1ed-b44e-464a-8144-8495dbcb15b7/volumes"
Nov 03 18:15:20.119708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.119694    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" path="/var/lib/kubelet/pods/711fa367-1899-4a42-9f96-cf28dbd0998e/volumes"
Nov 03 18:15:20.703297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.703264    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"
Nov 03 18:15:20.703297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.703273    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz" event={"ID":"3805976d-1f09-4603-ab4d-07154e278069","Type":"ContainerDied","Data":"25588a49ca929bbd6af8f0dfbfa3b3dde1cc1944fb6d7d112c4a7644f953679a"}
Nov 03 18:15:20.703870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.703344    5838 scope.go:122] "RemoveContainer" containerID="57024dffbc2517a0913e66ef0d249b38ffed1a6167e625799a12cd419b0b3e89"
Nov 03 18:15:20.872921 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.872881    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"]
Nov 03 18:15:20.879104 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:20.879072    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-qjtjz"]
Nov 03 18:15:21.564528 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:21.564491    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" podUID="7c40819a-98af-412e-8542-ccf40133be24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:22.117661 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:22.117613    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3805976d-1f09-4603-ab4d-07154e278069" path="/var/lib/kubelet/pods/3805976d-1f09-4603-ab4d-07154e278069/volumes"
Nov 03 18:15:24.789442 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:24.789385    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:25.214788 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:25.214714    5838 scope.go:122] "RemoveContainer" containerID="2ff935ba86b553eae8293029ef455c962a4ea1c64d48397018948dff34da164d"
Nov 03 18:15:27.947928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:27.947890    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:28.105807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:28.105771    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:28.272023 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:28.271938    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:28.428846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:28.428808    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:29.476610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.476575    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"]
Nov 03 18:15:29.477195 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477180    5838 cpu_manager.go:414] "containerMap: removing container" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:15:29.477241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477197    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:15:29.477241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477220    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:15:29.477241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477225    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477244    5838 cpu_manager.go:414] "containerMap: removing container" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477250    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477261    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477266    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477273    5838 cpu_manager.go:414] "containerMap: removing container" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477277    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477290    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477295    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477303    5838 cpu_manager.go:414] "containerMap: removing container" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477307    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477313    5838 cpu_manager.go:414] "containerMap: removing container" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477317    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477332    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:15:29.477382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477336    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477415    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477421    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477429    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477435    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477442    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477449    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477457    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:15:29.477875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.477462    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:15:29.483173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.483148    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.540217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.540178    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.540396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.540321    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcj5\" (UniqueName: \"kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.540396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.540366    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.540536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.540484    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.588307 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.588268    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0adf17-serving-cert\""
Nov 03 18:15:29.591528 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.591504    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"]
Nov 03 18:15:29.642244 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.642209    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcj5\" (UniqueName: \"kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.642476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.642252    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.642476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.642320    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.642476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.642392    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.642645 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:29.642548    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-serving-cert: secret "maas-api-e2e-inf-b-0adf17-serving-cert" not found
Nov 03 18:15:29.642645 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:29.642609    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls podName:1470c5b2-d2ad-4590-b08a-a41397b2b7b4 nodeName:}" failed. No retries permitted until 2026-11-03 18:15:30.142591423 +0000 UTC m=+2466.153329488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls") pod "maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" (UID: "1470c5b2-d2ad-4590-b08a-a41397b2b7b4") : secret "maas-api-e2e-inf-b-0adf17-serving-cert" not found
Nov 03 18:15:29.643119 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.643097    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.645232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.645208    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.676580 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.676545    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:29.696473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.696444    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:15:29.705601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.705573    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcj5\" (UniqueName: \"kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:29.839651 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.839554    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:29.994553 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:29.994524    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:30.153803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:30.153755    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:30.187927 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:30.186955    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:30.397660 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:30.397621    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:30.510072 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:30.510030    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:15:30.706874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:30.706789    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:15:30.783091 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:30.783059    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"]
Nov 03 18:15:30.784129 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:15:30.784106    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1470c5b2_d2ad_4590_b08a_a41397b2b7b4.slice/crio-293a4553e568ad6b0fa55a7ad83a070a088b2a364f44445560dbc9fa7be77eae WatchSource:0}: Error finding container 293a4553e568ad6b0fa55a7ad83a070a088b2a364f44445560dbc9fa7be77eae: Status 404 returned error can't find the container with id 293a4553e568ad6b0fa55a7ad83a070a088b2a364f44445560dbc9fa7be77eae
Nov 03 18:15:31.565000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:31.564945    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" podUID="7c40819a-98af-412e-8542-ccf40133be24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:31.801441 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:31.801098    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" event={"ID":"1470c5b2-d2ad-4590-b08a-a41397b2b7b4","Type":"ContainerStarted","Data":"2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c"}
Nov 03 18:15:31.801679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:31.801473    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" event={"ID":"1470c5b2-d2ad-4590-b08a-a41397b2b7b4","Type":"ContainerStarted","Data":"293a4553e568ad6b0fa55a7ad83a070a088b2a364f44445560dbc9fa7be77eae"}
Nov 03 18:15:31.801679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:31.801516    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:31.952596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:31.952526    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" podStartSLOduration=2.053462868 podStartE2EDuration="2.952507485s" totalImagesPullingTime="899.044617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:31.950205759 +0000 UTC m=+2467.960943830" watchObservedRunningTime="2026-11-03 18:15:31.952507485 +0000 UTC m=+2467.963245559"
Nov 03 18:15:35.003974 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:35.003932    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:35.549127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:35.549091    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:35.699781 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:35.699746    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:35.968770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:35.968704    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:37.828918 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:37.828890    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:15:38.782560 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.782474    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 18:15:38.873051 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.873015    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume\") pod \"7c40819a-98af-412e-8542-ccf40133be24\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") "
Nov 03 18:15:38.873545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.873145    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb\" (UniqueName: \"kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb\") pod \"7c40819a-98af-412e-8542-ccf40133be24\" (UID: \"7c40819a-98af-412e-8542-ccf40133be24\") "
Nov 03 18:15:38.873545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.873463    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume" pod "7c40819a-98af-412e-8542-ccf40133be24" (UID: "7c40819a-98af-412e-8542-ccf40133be24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:38.873716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.873697    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40819a-98af-412e-8542-ccf40133be24-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:38.875878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.875856    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb" pod "7c40819a-98af-412e-8542-ccf40133be24" (UID: "7c40819a-98af-412e-8542-ccf40133be24"). InnerVolumeSpecName "kube-api-access-69kjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:38.881354 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.881324    5838 generic.go:356] "Generic (PLEG): container finished" podID="7c40819a-98af-412e-8542-ccf40133be24" containerID="d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3" exitCode=0
Nov 03 18:15:38.881483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.881362    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"
Nov 03 18:15:38.881483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.881367    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" event={"ID":"7c40819a-98af-412e-8542-ccf40133be24","Type":"ContainerDied","Data":"d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3"}
Nov 03 18:15:38.881483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.881420    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn" event={"ID":"7c40819a-98af-412e-8542-ccf40133be24","Type":"ContainerDied","Data":"8f4acac1b3a03fcb37a1bcb8a624dda4aa4e5447ebf1a63c0dce6cc19ac95d90"}
Nov 03 18:15:38.881483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.881449    5838 scope.go:122] "RemoveContainer" containerID="d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3"
Nov 03 18:15:38.893510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.893491    5838 scope.go:122] "RemoveContainer" containerID="d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3"
Nov 03 18:15:38.893865 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:38.893845    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3\": container with ID starting with d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3 not found: ID does not exist" containerID="d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3"
Nov 03 18:15:38.893938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.893881    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3"} err="failed to get container status \"d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3\": rpc error: code = NotFound desc = could not find container \"d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3\": container with ID starting with d519ef6687c3dd1b652f957b30a7a4ecc60fb5fdf695cd71ace287d65e6b96a3 not found: ID does not exist"
Nov 03 18:15:38.975330 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:38.975289    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kjb\" (UniqueName: \"kubernetes.io/projected/7c40819a-98af-412e-8542-ccf40133be24-kube-api-access-69kjb\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:39.017970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:39.017936    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"]
Nov 03 18:15:39.024977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:39.024950    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-42c162-6fcd8f6875-prbqn"]
Nov 03 18:15:39.449058 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:39.449024    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:39.601230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:39.601196    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:40.117084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:40.117049    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c40819a-98af-412e-8542-ccf40133be24" path="/var/lib/kubelet/pods/7c40819a-98af-412e-8542-ccf40133be24/volumes"
Nov 03 18:15:40.134822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:40.134796    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:40.342269 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:15:40.342214    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:15:41.933427 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:41.933394    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:43.845716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:43.845674    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:44.895085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:44.895034    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:45.123083 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:45.122972    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:48.693526 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:48.693495    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:50.367892 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:50.367853    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:52.906459 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:52.906425    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:53.753215 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:53.753180    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:54.375480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:54.375444    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:54.616938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:54.616899    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:56.141925 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:56.141886    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:58.636043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:58.636002    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:58.792964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:58.792928    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:59.210225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:59.210167    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:59.410134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:15:59.410092    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.315086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.315042    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.318044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318014    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"]
Nov 03 18:16:01.318659 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318645    5838 cpu_manager.go:414] "containerMap: removing container" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:16:01.318705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318660    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:16:01.318705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318679    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:16:01.318705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318684    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318707    5838 cpu_manager.go:414] "containerMap: removing container" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318712    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318723    5838 cpu_manager.go:414] "containerMap: removing container" podUID="7c40819a-98af-412e-8542-ccf40133be24" containerName="payload-pre-processing"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318741    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c40819a-98af-412e-8542-ccf40133be24" containerName="payload-pre-processing"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318821    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="711fa367-1899-4a42-9f96-cf28dbd0998e" containerName="cleanup"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318829    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0842e288-6a80-4c7d-b7c7-0fe1a762d15c" containerName="cleanup"
Nov 03 18:16:01.318834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318837    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3805976d-1f09-4603-ab4d-07154e278069" containerName="cleanup"
Nov 03 18:16:01.319068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318849    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c40819a-98af-412e-8542-ccf40133be24" containerName="payload-pre-processing"
Nov 03 18:16:01.319068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.318857    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e32f1ed-b44e-464a-8144-8495dbcb15b7" containerName="cleanup"
Nov 03 18:16:01.322479 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.322458    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.356795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.356723    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.357037 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.357009    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.357155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.357075    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb89t\" (UniqueName: \"kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.357217 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.357152    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.383397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.383362    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"]
Nov 03 18:16:01.457835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.457802    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-44769b-serving-cert\""
Nov 03 18:16:01.458406 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.458371    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.458972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.458932    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb89t\" (UniqueName: \"kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.459261 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.459231    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.459468 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.459448    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.460348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.460323    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.460871 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:01.460849    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-serving-cert: secret "maas-api-e2e-api-b-44769b-serving-cert" not found
Nov 03 18:16:01.460969 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:01.460935    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls podName:50784f91-484c-499a-b13b-18acdcdc6c8a nodeName:}" failed. No retries permitted until 2026-11-03 18:16:01.960912041 +0000 UTC m=+2497.971650107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls") pod "maas-api-e2e-api-b-44769b-7949d8656d-lnp68" (UID: "50784f91-484c-499a-b13b-18acdcdc6c8a") : secret "maas-api-e2e-api-b-44769b-serving-cert" not found
Nov 03 18:16:01.464232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.464207    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.473035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.473009    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.525102 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.525054    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb89t\" (UniqueName: \"kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.638625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.638544    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.793085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.793053    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.953522 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.953485    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.966043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.966015    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:01.968709 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:01.968690    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\") pod \"maas-api-e2e-api-b-44769b-7949d8656d-lnp68\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:02.239113 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:02.239020    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:02.390496 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:02.390460    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:02.584753 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:02.584617    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:02.626617 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:02.626587    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"]
Nov 03 18:16:02.628066 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:16:02.628037    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50784f91_484c_499a_b13b_18acdcdc6c8a.slice/crio-2790010364e3e16d44c0b38b39c9ccad496be656564577678ca0537a872ce089 WatchSource:0}: Error finding container 2790010364e3e16d44c0b38b39c9ccad496be656564577678ca0537a872ce089: Status 404 returned error can't find the container with id 2790010364e3e16d44c0b38b39c9ccad496be656564577678ca0537a872ce089
Nov 03 18:16:03.059609 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:03.059566    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68" event={"ID":"50784f91-484c-499a-b13b-18acdcdc6c8a","Type":"ContainerStarted","Data":"2790010364e3e16d44c0b38b39c9ccad496be656564577678ca0537a872ce089"}
Nov 03 18:16:04.087035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:04.086984    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68" event={"ID":"50784f91-484c-499a-b13b-18acdcdc6c8a","Type":"ContainerStarted","Data":"5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62"}
Nov 03 18:16:04.087548 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:04.087113    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:04.210127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:04.210062    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68" podStartSLOduration=2.2991776010000002 podStartE2EDuration="3.210041995s" totalImagesPullingTime="910.864394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:04.207687723 +0000 UTC m=+2500.218425820" watchObservedRunningTime="2026-11-03 18:16:04.210041995 +0000 UTC m=+2500.220780069"
Nov 03 18:16:07.384663 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:07.384622    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:10.100885 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:10.100847    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:16:10.343796 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:10.343756    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:16:11.826286 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:11.826248    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:12.864519 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:12.864467    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:16.181433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:16.181393    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:16:18.110898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:18.110833    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:18.598991 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:18.598953    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:23.546591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:23.546540    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:23.740849 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:23.740814    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:28.862560 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:28.862524    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:30.459591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:30.459551    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"]
Nov 03 18:16:30.832024 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:30.831935    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:30.989244 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:30.989201    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:34.455159 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:34.455073    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:34.642114 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:34.642077    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:35.460631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:35.460547    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing" containerID="cri-o://968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef" gracePeriod=25
Nov 03 18:16:36.615746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:36.615684    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:16:39.596638 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:39.596604    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:40.341118 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:40.341068    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:16:44.013593 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.013558    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"]
Nov 03 18:16:44.014068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.013791    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" podUID="f754563c-befa-48c9-89a5-60a02450fd03" containerName="maas-api" containerID="cri-o://29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234" gracePeriod=30
Nov 03 18:16:44.175311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.175276    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:44.318793 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.318761    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 18:16:44.410794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.410699    5838 generic.go:356] "Generic (PLEG): container finished" podID="f754563c-befa-48c9-89a5-60a02450fd03" containerID="29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234" exitCode=0
Nov 03 18:16:44.411020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.410823    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" event={"ID":"f754563c-befa-48c9-89a5-60a02450fd03","Type":"ContainerDied","Data":"29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234"}
Nov 03 18:16:44.411020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.410990    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm" event={"ID":"f754563c-befa-48c9-89a5-60a02450fd03","Type":"ContainerDied","Data":"4081aef76fd0bf8043d7a8a8408849aee15bc7808deda7676b5a7cfada0fe278"}
Nov 03 18:16:44.411171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.411066    5838 scope.go:122] "RemoveContainer" containerID="29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234"
Nov 03 18:16:44.411618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.411595    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"
Nov 03 18:16:44.418484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.418339    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca\") pod \"f754563c-befa-48c9-89a5-60a02450fd03\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") "
Nov 03 18:16:44.418484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.418404    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs\") pod \"f754563c-befa-48c9-89a5-60a02450fd03\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") "
Nov 03 18:16:44.418484 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.418439    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4\" (UniqueName: \"kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4\") pod \"f754563c-befa-48c9-89a5-60a02450fd03\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") "
Nov 03 18:16:44.418700 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.418499    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\") pod \"f754563c-befa-48c9-89a5-60a02450fd03\" (UID: \"f754563c-befa-48c9-89a5-60a02450fd03\") "
Nov 03 18:16:44.419565 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.419418    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca" pod "f754563c-befa-48c9-89a5-60a02450fd03" (UID: "f754563c-befa-48c9-89a5-60a02450fd03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:44.428621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.428577    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4" pod "f754563c-befa-48c9-89a5-60a02450fd03" (UID: "f754563c-befa-48c9-89a5-60a02450fd03"). InnerVolumeSpecName "kube-api-access-p7ld4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:44.428902 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.428795    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls" pod "f754563c-befa-48c9-89a5-60a02450fd03" (UID: "f754563c-befa-48c9-89a5-60a02450fd03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:44.430913 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.430872    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs" pod "f754563c-befa-48c9-89a5-60a02450fd03" (UID: "f754563c-befa-48c9-89a5-60a02450fd03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:44.459442 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.459414    5838 scope.go:122] "RemoveContainer" containerID="29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234"
Nov 03 18:16:44.459951 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:44.459915    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234\": container with ID starting with 29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234 not found: ID does not exist" containerID="29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234"
Nov 03 18:16:44.460089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.459964    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234"} err="failed to get container status \"29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234\": rpc error: code = NotFound desc = could not find container \"29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234\": container with ID starting with 29822c37bf4fa750eb75ec1064c71612736283f96f77bd3802b3271b2869b234 not found: ID does not exist"
Nov 03 18:16:44.519831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.519788    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754563c-befa-48c9-89a5-60a02450fd03-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:44.519831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.519818    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:44.519831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.519828    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7ld4\" (UniqueName: \"kubernetes.io/projected/f754563c-befa-48c9-89a5-60a02450fd03-kube-api-access-p7ld4\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:44.519831 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.519839    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754563c-befa-48c9-89a5-60a02450fd03-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:44.838610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.838572    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"]
Nov 03 18:16:44.888768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:44.888697    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-548ca1-c7784b67d-c5xxm"]
Nov 03 18:16:45.301800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:45.301706    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:45.767650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:45.767607    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:46.002191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:46.002156    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:46.117856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:46.117770    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f754563c-befa-48c9-89a5-60a02450fd03" path="/var/lib/kubelet/pods/f754563c-befa-48c9-89a5-60a02450fd03/volumes"
Nov 03 18:16:46.160542 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:46.160503    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:46.616196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:46.616154    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:16:46.762548 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:46.762513    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:50.385692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:50.385655    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:50.606625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:50.606586    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"]
Nov 03 18:16:50.606851 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:50.606829    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" podUID="e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" containerName="maas-api" containerID="cri-o://8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185" gracePeriod=30
Nov 03 18:16:50.710468 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:50.710429    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:50.937678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:50.937648    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 18:16:51.005103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.005006    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca\") pod \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") "
Nov 03 18:16:51.005103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.005061    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\") pod \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") "
Nov 03 18:16:51.005103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.005092    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs\") pod \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") "
Nov 03 18:16:51.005424 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.005189    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx\" (UniqueName: \"kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx\") pod \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\" (UID: \"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a\") "
Nov 03 18:16:51.005491 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.005458    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca" pod "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" (UID: "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:51.005822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.005794    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:51.008774 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.008213    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls" pod "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" (UID: "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:51.018037 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.017947    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx" pod "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" (UID: "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a"). InnerVolumeSpecName "kube-api-access-zc7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:51.018601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.018564    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs" pod "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" (UID: "e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:51.106797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.106723    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:51.106797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.106799    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:51.106797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.106815    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7hx\" (UniqueName: \"kubernetes.io/projected/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a-kube-api-access-zc7hx\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:51.468974 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.468937    5838 generic.go:356] "Generic (PLEG): container finished" podID="e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" containerID="8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185" exitCode=0
Nov 03 18:16:51.469443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.468994    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"
Nov 03 18:16:51.469443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.469023    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" event={"ID":"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a","Type":"ContainerDied","Data":"8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185"}
Nov 03 18:16:51.469443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.469092    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh" event={"ID":"e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a","Type":"ContainerDied","Data":"ff3bead07a2bc45ff20b12166da0b86a58d237a19f021a5453247cad04c9df9d"}
Nov 03 18:16:51.469443 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.469122    5838 scope.go:122] "RemoveContainer" containerID="8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185"
Nov 03 18:16:51.480702 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.480679    5838 scope.go:122] "RemoveContainer" containerID="8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185"
Nov 03 18:16:51.481065 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:51.481044    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185\": container with ID starting with 8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185 not found: ID does not exist" containerID="8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185"
Nov 03 18:16:51.481123 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.481072    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185"} err="failed to get container status \"8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185\": rpc error: code = NotFound desc = could not find container \"8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185\": container with ID starting with 8ecb6f00ef2db12594b0b4198b574eb919043f21a3d0a3d928e87bec56e26185 not found: ID does not exist"
Nov 03 18:16:51.644941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.644885    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"]
Nov 03 18:16:51.650323 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:51.650295    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-42c162-fd7dc45f7-djtfh"]
Nov 03 18:16:52.118263 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:52.118225    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" path="/var/lib/kubelet/pods/e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a/volumes"
Nov 03 18:16:52.459942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:52.459900    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:52.725863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:52.725724    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:52.883238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:52.883207    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:53.463314 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:53.463277    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:54.996194 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:54.996163    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:55.063619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.063576    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:16:55.152433 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.152392    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:55.399036 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.399005    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:55.401366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.401335    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"]
Nov 03 18:16:55.401571 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.401550    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy" containerID="cri-o://e04a4894ffca0b36494777caf4d7b811bf3f01c21a3eb5be88068937dcdc0adf" gracePeriod=30
Nov 03 18:16:55.737333 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.737305    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 18:16:55.872075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.872037    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6\" (UniqueName: \"kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6\") pod \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") "
Nov 03 18:16:55.872280 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.872124    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume\") pod \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\" (UID: \"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8\") "
Nov 03 18:16:55.872564 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.872534    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume" pod "95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" (UID: "95793ffb-cf3e-4cb2-8a0e-b72aca7753c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:55.872725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.872703    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.874449 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.874419    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6" pod "95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" (UID: "95793ffb-cf3e-4cb2-8a0e-b72aca7753c8"). InnerVolumeSpecName "kube-api-access-2wrv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:55.974430 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:55.974344    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrv6\" (UniqueName: \"kubernetes.io/projected/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8-kube-api-access-2wrv6\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:56.512674 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.512629    5838 generic.go:356] "Generic (PLEG): container finished" podID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerID="968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef" exitCode=0
Nov 03 18:16:56.513132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.512696    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"
Nov 03 18:16:56.513132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.512720    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" event={"ID":"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8","Type":"ContainerDied","Data":"968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef"}
Nov 03 18:16:56.513132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.512805    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2" event={"ID":"95793ffb-cf3e-4cb2-8a0e-b72aca7753c8","Type":"ContainerDied","Data":"88e18ba68f9bdf7f343f3999369ea4d92369341804e71f5d47967c138fb90e6d"}
Nov 03 18:16:56.513132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.512827    5838 scope.go:122] "RemoveContainer" containerID="968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef"
Nov 03 18:16:56.523517 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.523495    5838 scope.go:122] "RemoveContainer" containerID="968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef"
Nov 03 18:16:56.523866 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:56.523827    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef\": container with ID starting with 968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef not found: ID does not exist" containerID="968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef"
Nov 03 18:16:56.523972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.523882    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef"} err="failed to get container status \"968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef\": rpc error: code = NotFound desc = could not find container \"968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef\": container with ID starting with 968e0266dd9655985ce758c8cb8ae27bcf6cd6b5a9756b1c374fc8c2c57293ef not found: ID does not exist"
Nov 03 18:16:56.682428 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.682384    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"]
Nov 03 18:16:56.688013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:56.687985    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c76280-5dcbd8f957-tf5g2"]
Nov 03 18:16:57.847884 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.847852    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"]
Nov 03 18:16:57.848665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848642    5838 cpu_manager.go:414] "containerMap: removing container" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing"
Nov 03 18:16:57.848787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848667    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing"
Nov 03 18:16:57.848787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848691    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" containerName="maas-api"
Nov 03 18:16:57.848787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848699    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" containerName="maas-api"
Nov 03 18:16:57.848787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848723    5838 cpu_manager.go:414] "containerMap: removing container" podUID="f754563c-befa-48c9-89a5-60a02450fd03" containerName="maas-api"
Nov 03 18:16:57.848787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848748    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f754563c-befa-48c9-89a5-60a02450fd03" containerName="maas-api"
Nov 03 18:16:57.849047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848884    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a4c0d4-4620-4d1c-874a-ac8f3ed84d2a" containerName="maas-api"
Nov 03 18:16:57.849047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848898    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f754563c-befa-48c9-89a5-60a02450fd03" containerName="maas-api"
Nov 03 18:16:57.849047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.848920    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" containerName="payload-processing"
Nov 03 18:16:57.853796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.853766    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:57.957674 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.957638    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ca3f31-serving-cert\""
Nov 03 18:16:57.960834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.960812    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"]
Nov 03 18:16:57.997856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.997822    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:57.998044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.997873    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgn94\" (UniqueName: \"kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:57.998044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.997894    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:57.998143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:57.998064    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.043796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.043765    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:58.101973 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.101849    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.101973 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.101910    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgn94\" (UniqueName: \"kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.101973 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.101941    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.102273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.102046    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.102273 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:58.102061    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-serving-cert: secret "maas-api-e2e-api-b-ca3f31-serving-cert" not found
Nov 03 18:16:58.102273 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:16:58.102151    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls podName:a367c84f-b8d1-43eb-8750-70918c5a5984 nodeName:}" failed. No retries permitted until 2026-11-03 18:16:58.602129526 +0000 UTC m=+2554.612867590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls") pod "maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" (UID: "a367c84f-b8d1-43eb-8750-70918c5a5984") : secret "maas-api-e2e-api-b-ca3f31-serving-cert" not found
Nov 03 18:16:58.102868 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.102839    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.105360 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.105322    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.120028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.119993    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95793ffb-cf3e-4cb2-8a0e-b72aca7753c8" path="/var/lib/kubelet/pods/95793ffb-cf3e-4cb2-8a0e-b72aca7753c8/volumes"
Nov 03 18:16:58.167070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.167037    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgn94\" (UniqueName: \"kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.212521 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.212488    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:58.367770 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.367671    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:58.609290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.609244    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.612063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.612038    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\") pod \"maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:58.768339 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:58.768287    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:16:59.160153 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:59.160116    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"]
Nov 03 18:16:59.161361 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:16:59.161330    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda367c84f_b8d1_43eb_8750_70918c5a5984.slice/crio-c5812cb94cd9e4843c901a6528226c0fae14de77827d02067b53ee8b86aca5df WatchSource:0}: Error finding container c5812cb94cd9e4843c901a6528226c0fae14de77827d02067b53ee8b86aca5df: Status 404 returned error can't find the container with id c5812cb94cd9e4843c901a6528226c0fae14de77827d02067b53ee8b86aca5df
Nov 03 18:16:59.539044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:16:59.538941    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" event={"ID":"a367c84f-b8d1-43eb-8750-70918c5a5984","Type":"ContainerStarted","Data":"c5812cb94cd9e4843c901a6528226c0fae14de77827d02067b53ee8b86aca5df"}
Nov 03 18:17:00.548067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.548032    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" event={"ID":"a367c84f-b8d1-43eb-8750-70918c5a5984","Type":"ContainerStarted","Data":"cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139"}
Nov 03 18:17:00.548565 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.548231    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:17:00.550529 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.550501    5838 generic.go:356] "Generic (PLEG): container finished" podID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerID="e04a4894ffca0b36494777caf4d7b811bf3f01c21a3eb5be88068937dcdc0adf" exitCode=0
Nov 03 18:17:00.550649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.550578    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" event={"ID":"9d79d5ae-44d1-4160-ae68-3e42df3a51a9","Type":"ContainerDied","Data":"e04a4894ffca0b36494777caf4d7b811bf3f01c21a3eb5be88068937dcdc0adf"}
Nov 03 18:17:00.550698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.550665    5838 scope.go:122] "RemoveContainer" containerID="88f2c9d88aa365f556b095f0bf6828a03a5116194128512b1838002729f80ad2"
Nov 03 18:17:00.671144 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.671039    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" podStartSLOduration=2.837171471 podStartE2EDuration="3.671022273s" totalImagesPullingTime="833.850802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:00.66848748 +0000 UTC m=+2556.679225561" watchObservedRunningTime="2026-11-03 18:17:00.671022273 +0000 UTC m=+2556.681760344"
Nov 03 18:17:00.688978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.688949    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 18:17:00.843989 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.843951    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.844257 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.843994    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.844257 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.844054    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.844450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.844321    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.844885 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.844835    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.844997 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.844905    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.844997 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.844984    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.845126 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.845062    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.845187 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.845149    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.845240 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.845182    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket\") pod \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\" (UID: \"9d79d5ae-44d1-4160-ae68-3e42df3a51a9\") "
Nov 03 18:17:00.846076 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.845915    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.846392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.846232    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.846474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.846441    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.846763 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.846718    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.846891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.846766    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.846891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.846789    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.846891 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.846805    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.847254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.847108    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:00.849528 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.849493    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.851205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.851174    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:00.851403 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.851374    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "kube-api-access-twl76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:00.852195 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.852167    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo" pod "9d79d5ae-44d1-4160-ae68-3e42df3a51a9" (UID: "9d79d5ae-44d1-4160-ae68-3e42df3a51a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:00.948332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.948220    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twl76\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-kube-api-access-twl76\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.948332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.948256    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.948332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.948270    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.948332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.948281    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.948332 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:00.948293    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d79d5ae-44d1-4160-ae68-3e42df3a51a9-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:01.558780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:01.558717    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"
Nov 03 18:17:01.558780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:01.558767    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5" event={"ID":"9d79d5ae-44d1-4160-ae68-3e42df3a51a9","Type":"ContainerDied","Data":"524071fef650ed5da5c8e175b370e5db8a482a46bd6e27fb3a09e28b953a0a3f"}
Nov 03 18:17:01.559357 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:01.558841    5838 scope.go:122] "RemoveContainer" containerID="e04a4894ffca0b36494777caf4d7b811bf3f01c21a3eb5be88068937dcdc0adf"
Nov 03 18:17:01.780626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:01.780585    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"]
Nov 03 18:17:01.830435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:01.830336    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-x8hq5"]
Nov 03 18:17:02.044853 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:02.044821    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:02.123345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:02.123261    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" path="/var/lib/kubelet/pods/9d79d5ae-44d1-4160-ae68-3e42df3a51a9/volumes"
Nov 03 18:17:02.205389 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:02.205353    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:02.484639 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:02.484607    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:03.391588 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:03.391554    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:03.856974 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:03.856898    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:04.021706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:04.021668    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:04.178303 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:04.178259    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:06.564771 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:06.564714    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:17:08.594560 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:08.594524    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:09.747792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:09.747755    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:10.339125 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:17:10.339081    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:17:11.260240 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:11.260197    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:13.109746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:13.109696    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:14.329474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:14.329443    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:15.880828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:15.880792    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:16.042678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:16.042648    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:17.302683 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:17.302650    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"]
Nov 03 18:17:17.303175 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:17.302871    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy" containerID="cri-o://9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d" gracePeriod=30
Nov 03 18:17:19.741877 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:19.741833    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:20.849946 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:20.849907    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:17:22.567473 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.567445    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 18:17:22.614134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614043    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614103    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614158    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614211    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614237    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614350    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614426    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614469    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614491    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:22.614621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614510    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert\") pod \"77397099-c52d-41db-b1ab-811d8b439547\" (UID: \"77397099-c52d-41db-b1ab-811d8b439547\") "
Nov 03 18:17:22.614621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614551    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:22.614968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.614716    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:22.615077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.615051    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.615077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.615079    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.615262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.615094    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.615262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.615142    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:22.615262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.615153    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:22.616575 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.616550    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:22.616678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.616633    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:22.617186 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.617164    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:22.617303 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.617189    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4" pod "77397099-c52d-41db-b1ab-811d8b439547" (UID: "77397099-c52d-41db-b1ab-811d8b439547"). InnerVolumeSpecName "kube-api-access-n29p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:22.716719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.716680    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.716719 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.716716    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29p4\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-kube-api-access-n29p4\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.716958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.716764    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77397099-c52d-41db-b1ab-811d8b439547-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.716958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.716775    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77397099-c52d-41db-b1ab-811d8b439547-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.716958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.716785    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77397099-c52d-41db-b1ab-811d8b439547-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.716958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.716795    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77397099-c52d-41db-b1ab-811d8b439547-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:22.719466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.719436    5838 generic.go:356] "Generic (PLEG): container finished" podID="77397099-c52d-41db-b1ab-811d8b439547" containerID="9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d" exitCode=0
Nov 03 18:17:22.719596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.719486    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"
Nov 03 18:17:22.719596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.719524    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" event={"ID":"77397099-c52d-41db-b1ab-811d8b439547","Type":"ContainerDied","Data":"9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d"}
Nov 03 18:17:22.719596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.719582    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh" event={"ID":"77397099-c52d-41db-b1ab-811d8b439547","Type":"ContainerDied","Data":"91eef89ecc76884cc75d6c97de6ab05d9da51aaa48da98c5cb695aad30f831c4"}
Nov 03 18:17:22.719809 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.719605    5838 scope.go:122] "RemoveContainer" containerID="9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d"
Nov 03 18:17:22.730785 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.730760    5838 scope.go:122] "RemoveContainer" containerID="9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d"
Nov 03 18:17:22.731102 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:17:22.731076    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d\": container with ID starting with 9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d not found: ID does not exist" containerID="9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d"
Nov 03 18:17:22.731170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.731114    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d"} err="failed to get container status \"9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d\": rpc error: code = NotFound desc = could not find container \"9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d\": container with ID starting with 9be1fea7464c2a70fde4c3ca8ed48590cabeb78e6b468ef0b6d5c76e3e0f7a0d not found: ID does not exist"
Nov 03 18:17:22.858597 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.858563    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"]
Nov 03 18:17:22.864510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:22.864430    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-ts4hh"]
Nov 03 18:17:24.119915 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:24.119881    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77397099-c52d-41db-b1ab-811d8b439547" path="/var/lib/kubelet/pods/77397099-c52d-41db-b1ab-811d8b439547/volumes"
Nov 03 18:17:24.187131 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:24.187099    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:24.343999 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:24.343955    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:25.125396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:25.125364    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:25.311566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:25.311534    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:29.580284 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:29.580245    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:30.280241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:30.280207    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:35.414740 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:35.414683    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:39.456375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:39.456339    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:39.617313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:39.617275    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:40.339168 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:17:40.339129    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:17:40.451764 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:40.451687    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:45.502935 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:45.502904    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:45.653536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:45.653503    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:50.628065 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:50.628028    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:51.830367 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:51.830333    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:56.227292 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:17:56.227255    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:00.710907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:00.710873    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:02.274576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:02.274511    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:02.494020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:02.493905    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:02.645607 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:02.645563    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:03.270423 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:03.270389    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:05.062992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:05.062955    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:05.215080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:05.215044    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:06.556966 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:06.556930    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:07.872416 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:07.872362    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:10.340381 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:18:10.340335    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:18:10.632936 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:10.632849    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:10.791797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:10.791759    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:11.042095 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:11.042061    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:12.294052 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:12.294021    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:18:12.880634 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:12.880584    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:17.590541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:17.590495    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:17.746317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:17.746277    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:17.904088 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:17.904050    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:22.923796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:22.923763    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:27.357635 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:27.357601    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:28.963652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:28.963616    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:29.184475 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:29.184431    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:29.336610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:29.336507    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:29.950136 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:29.950103    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:34.600556 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:34.600513    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:36.911184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:36.911131    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:37.096106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:37.096070    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:37.297575 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:37.297477    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:39.794329 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:39.794299    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:40.339607 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:18:40.339567    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:18:45.095655 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:45.095569    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:46.131787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:46.131744    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:48.822827 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:48.822790    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:18:49.727238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:49.727203    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"]
Nov 03 18:18:49.838336 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:49.838299    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"]
Nov 03 18:18:50.099929 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:50.099854    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:50.255456 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:50.255420    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:54.727577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:54.727514    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct" podUID="576d89a7-d224-40ec-9206-7cf0eb45fa93" containerName="payload-processing" containerID="cri-o://a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221" gracePeriod=25
Nov 03 18:18:54.839364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:54.839315    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt" podUID="645817b9-1596-4cf5-bf55-e33d0deb6658" containerName="payload-pre-processing" containerID="cri-o://e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54" gracePeriod=25
Nov 03 18:18:55.008150 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.008116    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:18:55.118998 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.118972    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:18:55.120942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.120916    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume\") pod \"576d89a7-d224-40ec-9206-7cf0eb45fa93\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") "
Nov 03 18:18:55.121119 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.121102    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r\" (UniqueName: \"kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r\") pod \"576d89a7-d224-40ec-9206-7cf0eb45fa93\" (UID: \"576d89a7-d224-40ec-9206-7cf0eb45fa93\") "
Nov 03 18:18:55.121369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.121343    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume" pod "576d89a7-d224-40ec-9206-7cf0eb45fa93" (UID: "576d89a7-d224-40ec-9206-7cf0eb45fa93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:55.121546 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.121529    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576d89a7-d224-40ec-9206-7cf0eb45fa93-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:55.123292 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.123268    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r" pod "576d89a7-d224-40ec-9206-7cf0eb45fa93" (UID: "576d89a7-d224-40ec-9206-7cf0eb45fa93"). InnerVolumeSpecName "kube-api-access-8ts8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:55.222846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.222793    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm\" (UniqueName: \"kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm\") pod \"645817b9-1596-4cf5-bf55-e33d0deb6658\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") "
Nov 03 18:18:55.223050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.222867    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume\") pod \"645817b9-1596-4cf5-bf55-e33d0deb6658\" (UID: \"645817b9-1596-4cf5-bf55-e33d0deb6658\") "
Nov 03 18:18:55.223287 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.223263    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume" pod "645817b9-1596-4cf5-bf55-e33d0deb6658" (UID: "645817b9-1596-4cf5-bf55-e33d0deb6658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:55.223432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.223414    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645817b9-1596-4cf5-bf55-e33d0deb6658-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:55.223483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.223440    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ts8r\" (UniqueName: \"kubernetes.io/projected/576d89a7-d224-40ec-9206-7cf0eb45fa93-kube-api-access-8ts8r\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:55.225308 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.225282    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm" pod "645817b9-1596-4cf5-bf55-e33d0deb6658" (UID: "645817b9-1596-4cf5-bf55-e33d0deb6658"). InnerVolumeSpecName "kube-api-access-wkwhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:55.252596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.249148    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:55.324554 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.324515    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwhm\" (UniqueName: \"kubernetes.io/projected/645817b9-1596-4cf5-bf55-e33d0deb6658-kube-api-access-wkwhm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:55.376846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.376810    5838 generic.go:356] "Generic (PLEG): container finished" podID="576d89a7-d224-40ec-9206-7cf0eb45fa93" containerID="a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221" exitCode=0
Nov 03 18:18:55.377050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.376863    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"
Nov 03 18:18:55.377050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.376905    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct" event={"ID":"576d89a7-d224-40ec-9206-7cf0eb45fa93","Type":"ContainerDied","Data":"a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221"}
Nov 03 18:18:55.377050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.376969    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct" event={"ID":"576d89a7-d224-40ec-9206-7cf0eb45fa93","Type":"ContainerDied","Data":"4932d38b0d74c585ab247f5d5cc3c0111c89bf77fa8b5b13c8eec38cfe8fece5"}
Nov 03 18:18:55.377050 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.376998    5838 scope.go:122] "RemoveContainer" containerID="a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221"
Nov 03 18:18:55.378546 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.378515    5838 generic.go:356] "Generic (PLEG): container finished" podID="645817b9-1596-4cf5-bf55-e33d0deb6658" containerID="e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54" exitCode=0
Nov 03 18:18:55.378679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.378561    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"
Nov 03 18:18:55.378679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.378556    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt" event={"ID":"645817b9-1596-4cf5-bf55-e33d0deb6658","Type":"ContainerDied","Data":"e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54"}
Nov 03 18:18:55.378840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.378701    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt" event={"ID":"645817b9-1596-4cf5-bf55-e33d0deb6658","Type":"ContainerDied","Data":"49ddeb9ffb3b38fee58d2cd77440ea26217dd0e49da467360a580a5891f9c78f"}
Nov 03 18:18:55.388834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.388810    5838 scope.go:122] "RemoveContainer" containerID="a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221"
Nov 03 18:18:55.389174 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:18:55.389154    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221\": container with ID starting with a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221 not found: ID does not exist" containerID="a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221"
Nov 03 18:18:55.389233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.389183    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221"} err="failed to get container status \"a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221\": rpc error: code = NotFound desc = could not find container \"a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221\": container with ID starting with a5983c74ef3c31ced8ee84d74bd4d03ae05e197780d09a141342169702648221 not found: ID does not exist"
Nov 03 18:18:55.389233 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.389210    5838 scope.go:122] "RemoveContainer" containerID="e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54"
Nov 03 18:18:55.398644 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.398620    5838 scope.go:122] "RemoveContainer" containerID="e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54"
Nov 03 18:18:55.398965 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:18:55.398943    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54\": container with ID starting with e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54 not found: ID does not exist" containerID="e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54"
Nov 03 18:18:55.399034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.398975    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54"} err="failed to get container status \"e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54\": rpc error: code = NotFound desc = could not find container \"e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54\": container with ID starting with e75440d27acce6802f1ea8c6f701f386c9678b3b9ca86a4c5cbe43b1ec9b7f54 not found: ID does not exist"
Nov 03 18:18:55.546006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.545918    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"]
Nov 03 18:18:55.552201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.552173    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44769b-69576fcb85-nxqct"]
Nov 03 18:18:55.664279 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.664247    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"]
Nov 03 18:18:55.672426 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:55.672396    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44769b-6b7ccbb6c9-6h6pt"]
Nov 03 18:18:56.118076 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:56.118036    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576d89a7-d224-40ec-9206-7cf0eb45fa93" path="/var/lib/kubelet/pods/576d89a7-d224-40ec-9206-7cf0eb45fa93/volumes"
Nov 03 18:18:56.118489 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:18:56.118435    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645817b9-1596-4cf5-bf55-e33d0deb6658" path="/var/lib/kubelet/pods/645817b9-1596-4cf5-bf55-e33d0deb6658/volumes"
Nov 03 18:19:00.279539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:00.279501    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:05.310501 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:05.310462    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:09.617463 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:09.617412    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"]
Nov 03 18:19:09.618010 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:09.617692    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" podUID="ee019581-201a-42e9-993d-dd262d39dfe7" containerName="maas-api" containerID="cri-o://7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b" gracePeriod=30
Nov 03 18:19:09.721380 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:09.721342    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:09.930359 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:09.930326    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:19:10.022178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.021187    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs\") pod \"ee019581-201a-42e9-993d-dd262d39dfe7\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") "
Nov 03 18:19:10.022178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.021275    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\") pod \"ee019581-201a-42e9-993d-dd262d39dfe7\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") "
Nov 03 18:19:10.022178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.021321    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht\" (UniqueName: \"kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht\") pod \"ee019581-201a-42e9-993d-dd262d39dfe7\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") "
Nov 03 18:19:10.022178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.021534    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca\") pod \"ee019581-201a-42e9-993d-dd262d39dfe7\" (UID: \"ee019581-201a-42e9-993d-dd262d39dfe7\") "
Nov 03 18:19:10.022758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.022713    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca" pod "ee019581-201a-42e9-993d-dd262d39dfe7" (UID: "ee019581-201a-42e9-993d-dd262d39dfe7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:10.030236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.030038    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs" pod "ee019581-201a-42e9-993d-dd262d39dfe7" (UID: "ee019581-201a-42e9-993d-dd262d39dfe7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:10.036200 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.036075    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht" pod "ee019581-201a-42e9-993d-dd262d39dfe7" (UID: "ee019581-201a-42e9-993d-dd262d39dfe7"). InnerVolumeSpecName "kube-api-access-blpht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:10.037888 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.037855    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls" pod "ee019581-201a-42e9-993d-dd262d39dfe7" (UID: "ee019581-201a-42e9-993d-dd262d39dfe7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:10.123013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.122973    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee019581-201a-42e9-993d-dd262d39dfe7-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:10.123013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.123009    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:10.123277 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.123025    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee019581-201a-42e9-993d-dd262d39dfe7-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:10.123277 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.123039    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blpht\" (UniqueName: \"kubernetes.io/projected/ee019581-201a-42e9-993d-dd262d39dfe7-kube-api-access-blpht\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:10.340430 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:19:10.340331    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:19:10.497011 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.496977    5838 generic.go:356] "Generic (PLEG): container finished" podID="ee019581-201a-42e9-993d-dd262d39dfe7" containerID="7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b" exitCode=0
Nov 03 18:19:10.497216 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.497034    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"
Nov 03 18:19:10.497216 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.497069    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" event={"ID":"ee019581-201a-42e9-993d-dd262d39dfe7","Type":"ContainerDied","Data":"7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b"}
Nov 03 18:19:10.497216 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.497139    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p" event={"ID":"ee019581-201a-42e9-993d-dd262d39dfe7","Type":"ContainerDied","Data":"06125d9d77380050191cfef9731ae712377973c2e49878e3e2303f1c9ab93ba4"}
Nov 03 18:19:10.497216 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.497172    5838 scope.go:122] "RemoveContainer" containerID="7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b"
Nov 03 18:19:10.509037 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.509011    5838 scope.go:122] "RemoveContainer" containerID="7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b"
Nov 03 18:19:10.509427 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:19:10.509401    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b\": container with ID starting with 7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b not found: ID does not exist" containerID="7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b"
Nov 03 18:19:10.509488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.509437    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b"} err="failed to get container status \"7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b\": rpc error: code = NotFound desc = could not find container \"7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b\": container with ID starting with 7b07c91e7e35fde8901ed5bde0c9074542bf8b825fd46b7d5b12627a313bad7b not found: ID does not exist"
Nov 03 18:19:10.668875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.668835    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"]
Nov 03 18:19:10.672042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:10.672018    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-43bdc0-6846f7fd85-snb2p"]
Nov 03 18:19:11.281760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:11.281698    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:11.511345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:11.511299    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:11.669582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:11.669542    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:12.118677 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:12.118568    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee019581-201a-42e9-993d-dd262d39dfe7" path="/var/lib/kubelet/pods/ee019581-201a-42e9-993d-dd262d39dfe7/volumes"
Nov 03 18:19:12.301967 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:12.301937    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:16.979271 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:16.979236    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:19.859512 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:19.859457    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:20.043832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:20.043777    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:20.316619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:20.316579    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:20.469902 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:20.469856    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:20.725662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:20.725622    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:22.151911 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:22.151858    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:24.695626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:24.695595    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:19:24.705846 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:24.705820    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:19:27.702002 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:27.701967    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:28.057543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:28.057438    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:19:32.715379 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:32.715278    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:33.563823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:33.563777    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:33.718016 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:33.717964    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:37.815219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:37.815173    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:40.339785 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:19:40.339718    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:19:42.862832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:42.862783    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:46.391047 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:46.390985    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:47.883237 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:47.883197    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:52.178045 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.178008    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"]
Nov 03 18:19:52.178577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.178274    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68" podUID="50784f91-484c-499a-b13b-18acdcdc6c8a" containerName="maas-api" containerID="cri-o://5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62" gracePeriod=30
Nov 03 18:19:52.277536 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.277496    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:52.469817 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.469790    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:19:52.636163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.636127    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca\") pod \"50784f91-484c-499a-b13b-18acdcdc6c8a\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") "
Nov 03 18:19:52.636397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.636197    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t\" (UniqueName: \"kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t\") pod \"50784f91-484c-499a-b13b-18acdcdc6c8a\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") "
Nov 03 18:19:52.636397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.636355    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs\") pod \"50784f91-484c-499a-b13b-18acdcdc6c8a\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") "
Nov 03 18:19:52.636397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.636388    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\") pod \"50784f91-484c-499a-b13b-18acdcdc6c8a\" (UID: \"50784f91-484c-499a-b13b-18acdcdc6c8a\") "
Nov 03 18:19:52.636640 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.636608    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca" pod "50784f91-484c-499a-b13b-18acdcdc6c8a" (UID: "50784f91-484c-499a-b13b-18acdcdc6c8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:52.639096 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.639051    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls" pod "50784f91-484c-499a-b13b-18acdcdc6c8a" (UID: "50784f91-484c-499a-b13b-18acdcdc6c8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:52.639268 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.639125    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t" pod "50784f91-484c-499a-b13b-18acdcdc6c8a" (UID: "50784f91-484c-499a-b13b-18acdcdc6c8a"). InnerVolumeSpecName "kube-api-access-kb89t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:52.639511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.639494    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs" pod "50784f91-484c-499a-b13b-18acdcdc6c8a" (UID: "50784f91-484c-499a-b13b-18acdcdc6c8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:52.738183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.738082    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:52.738183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.738127    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50784f91-484c-499a-b13b-18acdcdc6c8a-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:52.738183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.738143    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50784f91-484c-499a-b13b-18acdcdc6c8a-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:52.738183 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.738159    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb89t\" (UniqueName: \"kubernetes.io/projected/50784f91-484c-499a-b13b-18acdcdc6c8a-kube-api-access-kb89t\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:52.797513 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.797476    5838 generic.go:356] "Generic (PLEG): container finished" podID="50784f91-484c-499a-b13b-18acdcdc6c8a" containerID="5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62" exitCode=0
Nov 03 18:19:52.797696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.797547    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"
Nov 03 18:19:52.797696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.797576    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68" event={"ID":"50784f91-484c-499a-b13b-18acdcdc6c8a","Type":"ContainerDied","Data":"5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62"}
Nov 03 18:19:52.797696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.797645    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68" event={"ID":"50784f91-484c-499a-b13b-18acdcdc6c8a","Type":"ContainerDied","Data":"2790010364e3e16d44c0b38b39c9ccad496be656564577678ca0537a872ce089"}
Nov 03 18:19:52.797696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.797678    5838 scope.go:122] "RemoveContainer" containerID="5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62"
Nov 03 18:19:52.808323 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.808288    5838 scope.go:122] "RemoveContainer" containerID="5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62"
Nov 03 18:19:52.808672 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:19:52.808647    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62\": container with ID starting with 5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62 not found: ID does not exist" containerID="5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62"
Nov 03 18:19:52.808832 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.808683    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62"} err="failed to get container status \"5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62\": rpc error: code = NotFound desc = could not find container \"5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62\": container with ID starting with 5517470710ef03e3990442028c46debcfefb60d3fad7cd6e49c2fcaa7f1aaf62 not found: ID does not exist"
Nov 03 18:19:52.967713 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.967682    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"]
Nov 03 18:19:52.975721 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:52.975690    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44769b-7949d8656d-lnp68"]
Nov 03 18:19:53.864084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:53.864048    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:54.087497 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:54.087460    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:54.116953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:54.116871    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50784f91-484c-499a-b13b-18acdcdc6c8a" path="/var/lib/kubelet/pods/50784f91-484c-499a-b13b-18acdcdc6c8a/volumes"
Nov 03 18:19:54.242577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:54.242544    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:54.838413 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:54.838380    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:59.391290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:19:59.391252    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:02.185812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:02.185773    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:02.339457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:02.339426    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:02.568516 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:02.568417    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:03.057381 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:03.057345    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:04.582243 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:04.582203    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:05.255840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:05.255805    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:20:08.020792 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:08.020757    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"]
Nov 03 18:20:08.071103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:08.071067    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"]
Nov 03 18:20:08.190360 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:08.190323    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:09.899171 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:09.899135    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:10.340489 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:10.340406    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:20:14.885660 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:14.885627    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:17.529109 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:17.529069    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"]
Nov 03 18:20:17.638955 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:17.638917    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"]
Nov 03 18:20:17.894712 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:17.894681    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:18.047932 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:18.047898    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:19.997598 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:19.997552    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:22.530260 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:22.530208    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerName="payload-processing" containerID="cri-o://4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3" gracePeriod=25
Nov 03 18:20:22.639306 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:22.639261    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerName="payload-pre-processing" containerID="cri-o://f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d" gracePeriod=25
Nov 03 18:20:23.042659 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.042619    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main" containerID="cri-o://9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354" gracePeriod=45
Nov 03 18:20:23.092337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.092261    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main" containerID="cri-o://ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1" gracePeriod=45
Nov 03 18:20:23.379084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.379039    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 18:20:23.407070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.407034    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 18:20:23.504111 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504075    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm\") pod \"e84eff86-2993-4d78-a738-7d332dfbd042\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") "
Nov 03 18:20:23.504348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504175    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm\") pod \"d0b18255-4496-4e01-a9b5-2106b4493edd\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") "
Nov 03 18:20:23.504348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504208    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs\" (UniqueName: \"kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs\") pod \"e84eff86-2993-4d78-a738-7d332dfbd042\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") "
Nov 03 18:20:23.504348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504235    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6\" (UniqueName: \"kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6\") pod \"d0b18255-4496-4e01-a9b5-2106b4493edd\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") "
Nov 03 18:20:23.504348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504273    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs\") pod \"d0b18255-4496-4e01-a9b5-2106b4493edd\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") "
Nov 03 18:20:23.504592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504370    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf\" (UniqueName: \"kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf\") pod \"e84eff86-2993-4d78-a738-7d332dfbd042\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") "
Nov 03 18:20:23.504592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504418    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir\") pod \"e84eff86-2993-4d78-a738-7d332dfbd042\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") "
Nov 03 18:20:23.504592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504451    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache\") pod \"d0b18255-4496-4e01-a9b5-2106b4493edd\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") "
Nov 03 18:20:23.504592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504491    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home\") pod \"d0b18255-4496-4e01-a9b5-2106b4493edd\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") "
Nov 03 18:20:23.504592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504536    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache\") pod \"e84eff86-2993-4d78-a738-7d332dfbd042\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") "
Nov 03 18:20:23.504953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504607    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home\") pod \"e84eff86-2993-4d78-a738-7d332dfbd042\" (UID: \"e84eff86-2993-4d78-a738-7d332dfbd042\") "
Nov 03 18:20:23.504953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.504641    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir\") pod \"d0b18255-4496-4e01-a9b5-2106b4493edd\" (UID: \"d0b18255-4496-4e01-a9b5-2106b4493edd\") "
Nov 03 18:20:23.507566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.505187    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home" pod "d0b18255-4496-4e01-a9b5-2106b4493edd" (UID: "d0b18255-4496-4e01-a9b5-2106b4493edd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.507566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.505500    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir" pod "d0b18255-4496-4e01-a9b5-2106b4493edd" (UID: "d0b18255-4496-4e01-a9b5-2106b4493edd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.507566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.505701    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache" pod "e84eff86-2993-4d78-a738-7d332dfbd042" (UID: "e84eff86-2993-4d78-a738-7d332dfbd042"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.507566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.505929    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home" pod "e84eff86-2993-4d78-a738-7d332dfbd042" (UID: "e84eff86-2993-4d78-a738-7d332dfbd042"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.507566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.506100    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir" pod "e84eff86-2993-4d78-a738-7d332dfbd042" (UID: "e84eff86-2993-4d78-a738-7d332dfbd042"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.507566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.506283    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache" pod "d0b18255-4496-4e01-a9b5-2106b4493edd" (UID: "d0b18255-4496-4e01-a9b5-2106b4493edd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.509474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.508336    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6" pod "d0b18255-4496-4e01-a9b5-2106b4493edd" (UID: "d0b18255-4496-4e01-a9b5-2106b4493edd"). InnerVolumeSpecName "kube-api-access-2dnt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:23.509474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.508476    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs" pod "e84eff86-2993-4d78-a738-7d332dfbd042" (UID: "e84eff86-2993-4d78-a738-7d332dfbd042"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:23.509474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.508604    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm" pod "d0b18255-4496-4e01-a9b5-2106b4493edd" (UID: "d0b18255-4496-4e01-a9b5-2106b4493edd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.509474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.509154    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm" pod "e84eff86-2993-4d78-a738-7d332dfbd042" (UID: "e84eff86-2993-4d78-a738-7d332dfbd042"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.509474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.509243    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.509474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.509354    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf" pod "e84eff86-2993-4d78-a738-7d332dfbd042" (UID: "e84eff86-2993-4d78-a738-7d332dfbd042"). InnerVolumeSpecName "kube-api-access-sttlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:23.510598 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.510574    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs" pod "d0b18255-4496-4e01-a9b5-2106b4493edd" (UID: "d0b18255-4496-4e01-a9b5-2106b4493edd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610385    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610421    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610435    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610448    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610458    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610469    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b18255-4496-4e01-a9b5-2106b4493edd-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.610483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610479    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84eff86-2993-4d78-a738-7d332dfbd042-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.611141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610493    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnt6\" (UniqueName: \"kubernetes.io/projected/d0b18255-4496-4e01-a9b5-2106b4493edd-kube-api-access-2dnt6\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.611141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610507    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b18255-4496-4e01-a9b5-2106b4493edd-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.611141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610518    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttlf\" (UniqueName: \"kubernetes.io/projected/e84eff86-2993-4d78-a738-7d332dfbd042-kube-api-access-sttlf\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.611141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:23.610575    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84eff86-2993-4d78-a738-7d332dfbd042-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:24.036204 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.036168    5838 generic.go:356] "Generic (PLEG): container finished" podID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerID="9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354" exitCode=0
Nov 03 18:20:24.036452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.036212    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"
Nov 03 18:20:24.036452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.036261    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" event={"ID":"d0b18255-4496-4e01-a9b5-2106b4493edd","Type":"ContainerDied","Data":"9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354"}
Nov 03 18:20:24.036452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.036322    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn" event={"ID":"d0b18255-4496-4e01-a9b5-2106b4493edd","Type":"ContainerDied","Data":"8d714cff0ce08637843f214c5610e50e0f22e20ef14df1c2cab054a81bf8944e"}
Nov 03 18:20:24.036452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.036342    5838 scope.go:122] "RemoveContainer" containerID="9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354"
Nov 03 18:20:24.038110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.038070    5838 generic.go:356] "Generic (PLEG): container finished" podID="e84eff86-2993-4d78-a738-7d332dfbd042" containerID="ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1" exitCode=0
Nov 03 18:20:24.038247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.038113    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"
Nov 03 18:20:24.038247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.038159    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" event={"ID":"e84eff86-2993-4d78-a738-7d332dfbd042","Type":"ContainerDied","Data":"ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1"}
Nov 03 18:20:24.038247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.038204    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj" event={"ID":"e84eff86-2993-4d78-a738-7d332dfbd042","Type":"ContainerDied","Data":"f619654f55fe0445d2b4ca22158d507e0e624d46b9f73569b97fc80090bc6172"}
Nov 03 18:20:24.049765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.049724    5838 scope.go:122] "RemoveContainer" containerID="9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354"
Nov 03 18:20:24.050191 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:24.050161    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354\": container with ID starting with 9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354 not found: ID does not exist" containerID="9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354"
Nov 03 18:20:24.050336 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.050202    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354"} err="failed to get container status \"9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354\": rpc error: code = NotFound desc = could not find container \"9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354\": container with ID starting with 9d6877f77e2435485a13c962d4dc547fa753c022db266ce749bc3dc64658c354 not found: ID does not exist"
Nov 03 18:20:24.050336 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.050273    5838 scope.go:122] "RemoveContainer" containerID="ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1"
Nov 03 18:20:24.068447 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.068427    5838 scope.go:122] "RemoveContainer" containerID="ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1"
Nov 03 18:20:24.068852 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:24.068830    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1\": container with ID starting with ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1 not found: ID does not exist" containerID="ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1"
Nov 03 18:20:24.068926 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.068861    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1"} err="failed to get container status \"ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1\": rpc error: code = NotFound desc = could not find container \"ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1\": container with ID starting with ed03ebff69bfb993d1a30e256e30035a9850bb21ed76025f5311dad01be5eba1 not found: ID does not exist"
Nov 03 18:20:24.205776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.205725    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"]
Nov 03 18:20:24.212235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.212187    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-w6-f7d463-kserve-775f978ff5-fz8zj"]
Nov 03 18:20:24.290570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.290481    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"]
Nov 03 18:20:24.290754 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.290703    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" podUID="a367c84f-b8d1-43eb-8750-70918c5a5984" containerName="maas-api" containerID="cri-o://cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139" gracePeriod=30
Nov 03 18:20:24.329038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.328993    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"]
Nov 03 18:20:24.335342 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.335298    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/distinct-2-w6-f7d463-kserve-869b4754b8-znvtn"]
Nov 03 18:20:24.392500 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.392468    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:24.577251 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.577226    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:20:24.724976 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.724925    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94\" (UniqueName: \"kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94\") pod \"a367c84f-b8d1-43eb-8750-70918c5a5984\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") "
Nov 03 18:20:24.725480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.725001    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs\") pod \"a367c84f-b8d1-43eb-8750-70918c5a5984\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") "
Nov 03 18:20:24.725480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.725124    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca\") pod \"a367c84f-b8d1-43eb-8750-70918c5a5984\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") "
Nov 03 18:20:24.725480 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.725233    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\") pod \"a367c84f-b8d1-43eb-8750-70918c5a5984\" (UID: \"a367c84f-b8d1-43eb-8750-70918c5a5984\") "
Nov 03 18:20:24.727422 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.727050    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca" pod "a367c84f-b8d1-43eb-8750-70918c5a5984" (UID: "a367c84f-b8d1-43eb-8750-70918c5a5984"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:24.730043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.729877    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls" pod "a367c84f-b8d1-43eb-8750-70918c5a5984" (UID: "a367c84f-b8d1-43eb-8750-70918c5a5984"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:24.730043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.729911    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94" pod "a367c84f-b8d1-43eb-8750-70918c5a5984" (UID: "a367c84f-b8d1-43eb-8750-70918c5a5984"). InnerVolumeSpecName "kube-api-access-fgn94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:24.730343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.730321    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs" pod "a367c84f-b8d1-43eb-8750-70918c5a5984" (UID: "a367c84f-b8d1-43eb-8750-70918c5a5984"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:24.837823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.833534    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgn94\" (UniqueName: \"kubernetes.io/projected/a367c84f-b8d1-43eb-8750-70918c5a5984-kube-api-access-fgn94\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:24.837823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.833577    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:24.837823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.833592    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a367c84f-b8d1-43eb-8750-70918c5a5984-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:24.837823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:24.833606    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a367c84f-b8d1-43eb-8750-70918c5a5984-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:25.048265 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.048224    5838 generic.go:356] "Generic (PLEG): container finished" podID="a367c84f-b8d1-43eb-8750-70918c5a5984" containerID="cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139" exitCode=0
Nov 03 18:20:25.048488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.048284    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"
Nov 03 18:20:25.048488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.048317    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" event={"ID":"a367c84f-b8d1-43eb-8750-70918c5a5984","Type":"ContainerDied","Data":"cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139"}
Nov 03 18:20:25.048488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.048378    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh" event={"ID":"a367c84f-b8d1-43eb-8750-70918c5a5984","Type":"ContainerDied","Data":"c5812cb94cd9e4843c901a6528226c0fae14de77827d02067b53ee8b86aca5df"}
Nov 03 18:20:25.048488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.048407    5838 scope.go:122] "RemoveContainer" containerID="cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139"
Nov 03 18:20:25.063061 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.063042    5838 scope.go:122] "RemoveContainer" containerID="cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139"
Nov 03 18:20:25.063369 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:25.063350    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139\": container with ID starting with cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139 not found: ID does not exist" containerID="cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139"
Nov 03 18:20:25.063414 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.063380    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139"} err="failed to get container status \"cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139\": rpc error: code = NotFound desc = could not find container \"cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139\": container with ID starting with cca05d3986549bf195493ec516f26c29574b73b5e20f3082db284eca370c8139 not found: ID does not exist"
Nov 03 18:20:25.228388 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.228345    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"]
Nov 03 18:20:25.232445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:25.232402    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca3f31-d479d7dcf-sxmgh"]
Nov 03 18:20:26.096758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.096700    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:26.118182 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.118145    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a367c84f-b8d1-43eb-8750-70918c5a5984" path="/var/lib/kubelet/pods/a367c84f-b8d1-43eb-8750-70918c5a5984/volumes"
Nov 03 18:20:26.118824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.118804    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" path="/var/lib/kubelet/pods/d0b18255-4496-4e01-a9b5-2106b4493edd/volumes"
Nov 03 18:20:26.119358 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.119340    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" path="/var/lib/kubelet/pods/e84eff86-2993-4d78-a738-7d332dfbd042/volumes"
Nov 03 18:20:26.281090 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.281046    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:26.477509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.477477    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:26.531094 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.531053    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:20:26.531470 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.531443    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:20:26.979826 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:26.979786    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:28.076758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:28.076690    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:31.606231 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:31.606158    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:33.975779 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:33.975675    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:34.137141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:34.137108    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:34.366766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:34.366608    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:34.897841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:34.897788    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:36.530703 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:36.530667    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:20:36.531157 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:36.530666    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:20:36.801368 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:36.801262    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:37.330276 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.330230    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"]
Nov 03 18:20:37.330516 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.330487    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" podUID="a95322a2-de73-457e-baf4-293854dd05c1" containerName="discovery" containerID="cri-o://5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b" gracePeriod=30
Nov 03 18:20:37.604153 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.604122    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 18:20:37.692272 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692230    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.692481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692305    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.692481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692400    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.692481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692441    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.692611 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692549    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.692698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692665    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:37.692805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692690    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.692805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.692797    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token\") pod \"a95322a2-de73-457e-baf4-293854dd05c1\" (UID: \"a95322a2-de73-457e-baf4-293854dd05c1\") "
Nov 03 18:20:37.693289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.693242    5838 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-ca-configmap\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.695001 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.694974    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:37.695203 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.695167    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.695316 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.695275    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.695316 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.695271    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:37.695546 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.695529    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "kube-api-access-6nfck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.695777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.695761    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts" pod "a95322a2-de73-457e-baf4-293854dd05c1" (UID: "a95322a2-de73-457e-baf4-293854dd05c1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:37.794649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.794616    5838 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-csr-dns-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.794649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.794648    5838 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95322a2-de73-457e-baf4-293854dd05c1-local-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.794907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.794657    5838 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-cacerts\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.794907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.794668    5838 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a95322a2-de73-457e-baf4-293854dd05c1-istio-kubeconfig\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.794907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.794679    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfck\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-kube-api-access-6nfck\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.794907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:37.794688    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95322a2-de73-457e-baf4-293854dd05c1-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:38.140160 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.140118    5838 generic.go:356] "Generic (PLEG): container finished" podID="a95322a2-de73-457e-baf4-293854dd05c1" containerID="5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b" exitCode=0
Nov 03 18:20:38.140160 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.140152    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" event={"ID":"a95322a2-de73-457e-baf4-293854dd05c1","Type":"ContainerDied","Data":"5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b"}
Nov 03 18:20:38.140474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.140217    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4" event={"ID":"a95322a2-de73-457e-baf4-293854dd05c1","Type":"ContainerDied","Data":"4e919a6770cda06d767e5d05e1137c934c16c89be41eb3ead14736cb3e7088ee"}
Nov 03 18:20:38.140474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.140237    5838 scope.go:122] "RemoveContainer" containerID="5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b"
Nov 03 18:20:38.140474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.140167    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"
Nov 03 18:20:38.151034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.151006    5838 scope.go:122] "RemoveContainer" containerID="5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b"
Nov 03 18:20:38.151362 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:38.151339    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b\": container with ID starting with 5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b not found: ID does not exist" containerID="5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b"
Nov 03 18:20:38.151436 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.151372    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b"} err="failed to get container status \"5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b\": rpc error: code = NotFound desc = could not find container \"5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b\": container with ID starting with 5021085165d8c5d152ab0b4db2e25dcfb1c5a6ba7a2514b5d7307de8563eaa1b not found: ID does not exist"
Nov 03 18:20:38.313001 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.312969    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"]
Nov 03 18:20:38.363168 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:38.363130    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6m8b4"]
Nov 03 18:20:39.859082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:39.859050    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"]
Nov 03 18:20:39.991676 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:39.991635    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:40.118063 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:40.117975    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95322a2-de73-457e-baf4-293854dd05c1" path="/var/lib/kubelet/pods/a95322a2-de73-457e-baf4-293854dd05c1/volumes"
Nov 03 18:20:40.344899 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:40.344858    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:20:42.059493 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.059441    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:42.561008 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.560981    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:20:42.806343 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.806274    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 18:20:42.866410 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.865473    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285\" (UniqueName: \"kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285\") pod \"0c4d401c-e67d-444d-9391-0d36da0c9db4\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") "
Nov 03 18:20:42.866410 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.865699    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume\") pod \"0c4d401c-e67d-444d-9391-0d36da0c9db4\" (UID: \"0c4d401c-e67d-444d-9391-0d36da0c9db4\") "
Nov 03 18:20:42.866886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.866845    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume" pod "0c4d401c-e67d-444d-9391-0d36da0c9db4" (UID: "0c4d401c-e67d-444d-9391-0d36da0c9db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:42.871188 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.871153    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285" pod "0c4d401c-e67d-444d-9391-0d36da0c9db4" (UID: "0c4d401c-e67d-444d-9391-0d36da0c9db4"). InnerVolumeSpecName "kube-api-access-kd285". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:42.920766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.920726    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 18:20:42.967262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.967224    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4d401c-e67d-444d-9391-0d36da0c9db4-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:42.967262 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:42.967259    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd285\" (UniqueName: \"kubernetes.io/projected/0c4d401c-e67d-444d-9391-0d36da0c9db4-kube-api-access-kd285\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.068509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.068406    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume\") pod \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") "
Nov 03 18:20:43.068509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.068492    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn\" (UniqueName: \"kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn\") pod \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\" (UID: \"f037b0ba-514f-4ca7-a1b7-5943f29ae94d\") "
Nov 03 18:20:43.070040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.070008    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume" pod "f037b0ba-514f-4ca7-a1b7-5943f29ae94d" (UID: "f037b0ba-514f-4ca7-a1b7-5943f29ae94d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:43.072100 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.072073    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn" pod "f037b0ba-514f-4ca7-a1b7-5943f29ae94d" (UID: "f037b0ba-514f-4ca7-a1b7-5943f29ae94d"). InnerVolumeSpecName "kube-api-access-xvpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:43.170228 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.170186    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.170228 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.170221    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvpgn\" (UniqueName: \"kubernetes.io/projected/f037b0ba-514f-4ca7-a1b7-5943f29ae94d-kube-api-access-xvpgn\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.175311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.175277    5838 generic.go:356] "Generic (PLEG): container finished" podID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerID="4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3" exitCode=0
Nov 03 18:20:43.175507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.175330    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"
Nov 03 18:20:43.175507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.175356    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" event={"ID":"0c4d401c-e67d-444d-9391-0d36da0c9db4","Type":"ContainerDied","Data":"4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3"}
Nov 03 18:20:43.175507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.175416    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d" event={"ID":"0c4d401c-e67d-444d-9391-0d36da0c9db4","Type":"ContainerDied","Data":"23f04bebe3e04bb1a37a91bab60034811bf8b140e50580d7262edc28b7bf8370"}
Nov 03 18:20:43.175507 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.175440    5838 scope.go:122] "RemoveContainer" containerID="4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3"
Nov 03 18:20:43.177121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.177098    5838 generic.go:356] "Generic (PLEG): container finished" podID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerID="f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d" exitCode=0
Nov 03 18:20:43.177274 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.177146    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"
Nov 03 18:20:43.177274 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.177160    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" event={"ID":"f037b0ba-514f-4ca7-a1b7-5943f29ae94d","Type":"ContainerDied","Data":"f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d"}
Nov 03 18:20:43.177274 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.177193    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2" event={"ID":"f037b0ba-514f-4ca7-a1b7-5943f29ae94d","Type":"ContainerDied","Data":"a8f512c9b3db9848b6922bbad003e45489c44df89c28aceabc459fd580e2500e"}
Nov 03 18:20:43.186977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.186960    5838 scope.go:122] "RemoveContainer" containerID="4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3"
Nov 03 18:20:43.187432 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:43.187308    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3\": container with ID starting with 4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3 not found: ID does not exist" containerID="4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3"
Nov 03 18:20:43.187432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.187334    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3"} err="failed to get container status \"4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3\": rpc error: code = NotFound desc = could not find container \"4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3\": container with ID starting with 4c8a0d8f7f1c34ba81cbb332470e007e9455f06c08ff710aa38c7b0a9fe4bde3 not found: ID does not exist"
Nov 03 18:20:43.187432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.187353    5838 scope.go:122] "RemoveContainer" containerID="f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d"
Nov 03 18:20:43.198195 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.198166    5838 scope.go:122] "RemoveContainer" containerID="f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d"
Nov 03 18:20:43.198561 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:43.198540    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d\": container with ID starting with f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d not found: ID does not exist" containerID="f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d"
Nov 03 18:20:43.198634 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.198574    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d"} err="failed to get container status \"f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d\": rpc error: code = NotFound desc = could not find container \"f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d\": container with ID starting with f5ce1e708c8e30aeb9279e520515b495d128687d694b284dbfd2b754b6f8cb5d not found: ID does not exist"
Nov 03 18:20:43.347368 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.347273    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"]
Nov 03 18:20:43.353978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.353950    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7d463-9484db4ff-6rn8d"]
Nov 03 18:20:43.466322 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.466285    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"]
Nov 03 18:20:43.473615 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:43.473583    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7d463-f94d5db95-8l5s2"]
Nov 03 18:20:44.116408 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:44.116373    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" path="/var/lib/kubelet/pods/0c4d401c-e67d-444d-9391-0d36da0c9db4/volumes"
Nov 03 18:20:44.116835 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:44.116807    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" path="/var/lib/kubelet/pods/f037b0ba-514f-4ca7-a1b7-5943f29ae94d/volumes"
Nov 03 18:20:46.780096 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:46.780041    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:47.331027 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:47.330991    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:47.483598 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:47.483544    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:47.644842 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:47.644801    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:49.062205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:49.062103    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:49.265466 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:49.265416    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:49.464963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:49.464932    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:51.752574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:51.752536    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:53.367336 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:53.367300    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:53.589599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:53.589557    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:53.746572 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:53.746521    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:54.356998 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:54.356961    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:54.880970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:54.880923    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" containerID="cri-o://765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26" gracePeriod=45
Nov 03 18:20:55.010858 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.010812    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused"
Nov 03 18:20:55.154942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.154916    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 18:20:55.256960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.256917    5838 generic.go:356] "Generic (PLEG): container finished" podID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerID="765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26" exitCode=0
Nov 03 18:20:55.256960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.256964    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"
Nov 03 18:20:55.257179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.257002    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" event={"ID":"98b8fa95-a4fa-4e85-a88e-ac177e5cb602","Type":"ContainerDied","Data":"765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26"}
Nov 03 18:20:55.257179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.257052    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46" event={"ID":"98b8fa95-a4fa-4e85-a88e-ac177e5cb602","Type":"ContainerDied","Data":"74a71553ef253daf65589b54fedf72ef94aad90fb72eb890ecabc4f69a04bce0"}
Nov 03 18:20:55.257179 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.257069    5838 scope.go:122] "RemoveContainer" containerID="765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26"
Nov 03 18:20:55.269402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.269379    5838 scope.go:122] "RemoveContainer" containerID="765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26"
Nov 03 18:20:55.269726 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:20:55.269705    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26\": container with ID starting with 765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26 not found: ID does not exist" containerID="765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26"
Nov 03 18:20:55.269833 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.269754    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26"} err="failed to get container status \"765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26\": rpc error: code = NotFound desc = could not find container \"765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26\": container with ID starting with 765b8d567418e001716b9b74168eee8226887f8da6a01df54c0205a27048ec26 not found: ID does not exist"
Nov 03 18:20:55.303257 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303218    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache\") pod \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") "
Nov 03 18:20:55.303467 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303275    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs\" (UniqueName: \"kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs\") pod \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") "
Nov 03 18:20:55.303467 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303325    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir\") pod \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") "
Nov 03 18:20:55.303467 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303355    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home\") pod \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") "
Nov 03 18:20:55.303467 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303406    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v\" (UniqueName: \"kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v\") pod \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") "
Nov 03 18:20:55.303467 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303439    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm\") pod \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\" (UID: \"98b8fa95-a4fa-4e85-a88e-ac177e5cb602\") "
Nov 03 18:20:55.303768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303545    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache" pod "98b8fa95-a4fa-4e85-a88e-ac177e5cb602" (UID: "98b8fa95-a4fa-4e85-a88e-ac177e5cb602"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.303768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303567    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir" pod "98b8fa95-a4fa-4e85-a88e-ac177e5cb602" (UID: "98b8fa95-a4fa-4e85-a88e-ac177e5cb602"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.303768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303676    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home" pod "98b8fa95-a4fa-4e85-a88e-ac177e5cb602" (UID: "98b8fa95-a4fa-4e85-a88e-ac177e5cb602"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.303937 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303890    5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-model-cache\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.303937 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303916    5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tmp-dir\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.303937 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.303931    5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-home\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.305960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.305934    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs" pod "98b8fa95-a4fa-4e85-a88e-ac177e5cb602" (UID: "98b8fa95-a4fa-4e85-a88e-ac177e5cb602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:55.305960 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.305944    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm" pod "98b8fa95-a4fa-4e85-a88e-ac177e5cb602" (UID: "98b8fa95-a4fa-4e85-a88e-ac177e5cb602"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.306182 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.306045    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v" pod "98b8fa95-a4fa-4e85-a88e-ac177e5cb602" (UID: "98b8fa95-a4fa-4e85-a88e-ac177e5cb602"). InnerVolumeSpecName "kube-api-access-mzf7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:55.404660 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.404626    5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-dshm\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.404660 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.404659    5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-tls-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.404895 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.404669    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzf7v\" (UniqueName: \"kubernetes.io/projected/98b8fa95-a4fa-4e85-a88e-ac177e5cb602-kube-api-access-mzf7v\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.681922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.681832    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"]
Nov 03 18:20:55.732130 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:55.732088    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-2-w4-73a606-kserve-856f677f69-d8t46"]
Nov 03 18:20:56.116041 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:56.115955    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" path="/var/lib/kubelet/pods/98b8fa95-a4fa-4e85-a88e-ac177e5cb602/volumes"
Nov 03 18:20:56.638562 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:56.638520    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:58.898627 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:20:58.898590    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:02.286856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:02.286814    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:03.177765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.177701    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:03.332711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.332661    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:03.476889 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.476794    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"]
Nov 03 18:21:03.477402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477381    5838 cpu_manager.go:414] "containerMap: removing container" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477404    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477438    5838 cpu_manager.go:414] "containerMap: removing container" podUID="645817b9-1596-4cf5-bf55-e33d0deb6658" containerName="payload-pre-processing"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477448    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645817b9-1596-4cf5-bf55-e33d0deb6658" containerName="payload-pre-processing"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477460    5838 cpu_manager.go:414] "containerMap: removing container" podUID="576d89a7-d224-40ec-9206-7cf0eb45fa93" containerName="payload-processing"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477468    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576d89a7-d224-40ec-9206-7cf0eb45fa93" containerName="payload-processing"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477479    5838 cpu_manager.go:414] "containerMap: removing container" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477486    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477507    5838 cpu_manager.go:414] "containerMap: removing container" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy"
Nov 03 18:21:03.477520 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477514    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477525    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a367c84f-b8d1-43eb-8750-70918c5a5984" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477531    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a367c84f-b8d1-43eb-8750-70918c5a5984" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477547    5838 cpu_manager.go:414] "containerMap: removing container" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477554    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477567    5838 cpu_manager.go:414] "containerMap: removing container" podUID="50784f91-484c-499a-b13b-18acdcdc6c8a" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477574    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50784f91-484c-499a-b13b-18acdcdc6c8a" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477585    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a95322a2-de73-457e-baf4-293854dd05c1" containerName="discovery"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477592    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95322a2-de73-457e-baf4-293854dd05c1" containerName="discovery"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477610    5838 cpu_manager.go:414] "containerMap: removing container" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerName="payload-pre-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477616    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerName="payload-pre-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477627    5838 cpu_manager.go:414] "containerMap: removing container" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerName="payload-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477634    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerName="payload-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477644    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ee019581-201a-42e9-993d-dd262d39dfe7" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477651    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee019581-201a-42e9-993d-dd262d39dfe7" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477674    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477681    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477834    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="645817b9-1596-4cf5-bf55-e33d0deb6658" containerName="payload-pre-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477848    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="576d89a7-d224-40ec-9206-7cf0eb45fa93" containerName="payload-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477858    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477870    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="77397099-c52d-41db-b1ab-811d8b439547" containerName="istio-proxy"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477882    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a367c84f-b8d1-43eb-8750-70918c5a5984" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477894    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b18255-4496-4e01-a9b5-2106b4493edd" containerName="main"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477903    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="50784f91-484c-499a-b13b-18acdcdc6c8a" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477913    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95322a2-de73-457e-baf4-293854dd05c1" containerName="discovery"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477922    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f037b0ba-514f-4ca7-a1b7-5943f29ae94d" containerName="payload-pre-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477932    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4d401c-e67d-444d-9391-0d36da0c9db4" containerName="payload-processing"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477942    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee019581-201a-42e9-993d-dd262d39dfe7" containerName="maas-api"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477955    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84eff86-2993-4d78-a738-7d332dfbd042" containerName="main"
Nov 03 18:21:03.478020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.477966    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b8fa95-a4fa-4e85-a88e-ac177e5cb602" containerName="main"
Nov 03 18:21:03.483371 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.483340    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.489386 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.489354    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shr5d\" (UniqueName: \"kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.489563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.489483    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.489625 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.489597    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.543091 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.543054    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"]
Nov 03 18:21:03.578197 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.578158    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:03.583574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.583540    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:21:03.590511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.590483    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:21:03.590771 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.590710    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.590870 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.590818    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.590951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.590924    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shr5d\" (UniqueName: \"kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.591677 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.591650    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.594176 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.594150    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.657083 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.657053    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shr5d\" (UniqueName: \"kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d\") pod \"maas-api-revoke-sub-cc0125bb-vdklv\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:03.805275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:03.805188    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:04.192942 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:04.192909    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"]
Nov 03 18:21:04.195157 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:21:04.195126    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08da3b48_fdc8_4487_a7ce_7990cdec7c6a.slice/crio-3cdc2b982854e23d605fcdbbb31eba3605846a5a946f7848283279096d137eac WatchSource:0}: Error finding container 3cdc2b982854e23d605fcdbbb31eba3605846a5a946f7848283279096d137eac: Status 404 returned error can't find the container with id 3cdc2b982854e23d605fcdbbb31eba3605846a5a946f7848283279096d137eac
Nov 03 18:21:04.319075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:04.318830    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv" event={"ID":"08da3b48-fdc8-4487-a7ce-7990cdec7c6a","Type":"ContainerStarted","Data":"3cdc2b982854e23d605fcdbbb31eba3605846a5a946f7848283279096d137eac"}
Nov 03 18:21:04.612068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:04.612030    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:05.311996 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:05.311943    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:05.326964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:05.326931    5838 generic.go:356] "Generic (PLEG): container finished" podID="08da3b48-fdc8-4487-a7ce-7990cdec7c6a" containerID="787ae5cb66fc21a69faf6312f054e1ad5c29f3ad249f0c7f88241873b71f4471" exitCode=0
Nov 03 18:21:05.327160 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:05.327020    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv" event={"ID":"08da3b48-fdc8-4487-a7ce-7990cdec7c6a","Type":"ContainerDied","Data":"787ae5cb66fc21a69faf6312f054e1ad5c29f3ad249f0c7f88241873b71f4471"}
Nov 03 18:21:05.466857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:05.466682    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:06.478699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.478671    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:06.519882 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.519849    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d\" (UniqueName: \"kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d\") pod \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") "
Nov 03 18:21:06.520068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.519931    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca\") pod \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") "
Nov 03 18:21:06.520068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.519976    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca\") pod \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\" (UID: \"08da3b48-fdc8-4487-a7ce-7990cdec7c6a\") "
Nov 03 18:21:06.520369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.520342    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca" pod "08da3b48-fdc8-4487-a7ce-7990cdec7c6a" (UID: "08da3b48-fdc8-4487-a7ce-7990cdec7c6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:06.522383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.522359    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca" pod "08da3b48-fdc8-4487-a7ce-7990cdec7c6a" (UID: "08da3b48-fdc8-4487-a7ce-7990cdec7c6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:21:06.522509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.522493    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d" pod "08da3b48-fdc8-4487-a7ce-7990cdec7c6a" (UID: "08da3b48-fdc8-4487-a7ce-7990cdec7c6a"). InnerVolumeSpecName "kube-api-access-shr5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:06.621685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.621601    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shr5d\" (UniqueName: \"kubernetes.io/projected/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-kube-api-access-shr5d\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.621685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.621637    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-openshift-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.621685 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:06.621648    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08da3b48-fdc8-4487-a7ce-7990cdec7c6a-maas-api-xks-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:07.341887 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:07.341842    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv" event={"ID":"08da3b48-fdc8-4487-a7ce-7990cdec7c6a","Type":"ContainerDied","Data":"3cdc2b982854e23d605fcdbbb31eba3605846a5a946f7848283279096d137eac"}
Nov 03 18:21:07.342085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:07.341906    5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cdc2b982854e23d605fcdbbb31eba3605846a5a946f7848283279096d137eac"
Nov 03 18:21:07.342085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:07.341858    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"
Nov 03 18:21:09.998194 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:09.998158    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:10.340082 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:21:10.339976    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:21:14.793178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:14.793146    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:14.947749 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:14.947681    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:15.103260 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:15.103175    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:19.547619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:19.547552    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:19.755078 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:19.755037    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:20.085278 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:20.085247    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:20.238509 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:20.238459    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:24.524096 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:24.524058    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:26.082539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:26.082500    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:26.313328 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:26.313274    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:26.468349 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:26.468319    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:26.822411 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:26.822303    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:27.077564 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:27.077445    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:31.164267 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:31.164227    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:32.761937 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:32.761896    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:33.012803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:33.012633    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:33.169545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:33.169508    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:33.746165 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:33.746129    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:34.172092 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:34.172057    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:34.324324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:34.324280    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:34.932142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:34.932027    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:35.085040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:35.085011    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:35.307643 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:35.307565    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:37.838659 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:37.838616    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:21:38.411127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:38.411096    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:40.339185 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:21:40.339146    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:21:41.755626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:41.755593    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:41.908256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:41.908221    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:42.136711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:42.136604    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:43.647163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:43.647129    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:47.124574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:47.124535    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:47.280840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:47.280802    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:49.210375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:49.210343    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:54.275381 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:54.275346    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:54.626090 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:54.626001    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"]
Nov 03 18:21:54.739235 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:54.739198    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"]
Nov 03 18:21:55.001784 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:55.001747    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:55.168244 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:55.168204    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:59.289687 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:59.289652    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:59.626419 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:59.626324    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" containerName="payload-processing" containerID="cri-o://2421720b4c818d5a14b89348d9678f062a08e86af9639388167426e21ce3ab4e" gracePeriod=25
Nov 03 18:21:59.739750 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:21:59.739676    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" containerName="payload-pre-processing" containerID="cri-o://17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d" gracePeriod=25
Nov 03 18:22:03.617800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:03.617764    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"]
Nov 03 18:22:03.618287 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:03.617976    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" podUID="1470c5b2-d2ad-4590-b08a-a41397b2b7b4" containerName="maas-api" containerID="cri-o://2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c" gracePeriod=30
Nov 03 18:22:03.720226 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:03.720190    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:03.887085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:03.887056    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:22:04.037469 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.037425    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca\") pod \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") "
Nov 03 18:22:04.037667 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.037545    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs\") pod \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") "
Nov 03 18:22:04.037776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.037679    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\") pod \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") "
Nov 03 18:22:04.037776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.037760    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5\" (UniqueName: \"kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5\") pod \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\" (UID: \"1470c5b2-d2ad-4590-b08a-a41397b2b7b4\") "
Nov 03 18:22:04.037930 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.037896    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca" pod "1470c5b2-d2ad-4590-b08a-a41397b2b7b4" (UID: "1470c5b2-d2ad-4590-b08a-a41397b2b7b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:04.038211 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.038183    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:04.039975 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.039946    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs" pod "1470c5b2-d2ad-4590-b08a-a41397b2b7b4" (UID: "1470c5b2-d2ad-4590-b08a-a41397b2b7b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:04.040519 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.040486    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls" pod "1470c5b2-d2ad-4590-b08a-a41397b2b7b4" (UID: "1470c5b2-d2ad-4590-b08a-a41397b2b7b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:04.040610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.040523    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5" pod "1470c5b2-d2ad-4590-b08a-a41397b2b7b4" (UID: "1470c5b2-d2ad-4590-b08a-a41397b2b7b4"). InnerVolumeSpecName "kube-api-access-swcj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:04.139275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.139241    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:04.139275 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.139274    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:04.139489 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.139284    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swcj5\" (UniqueName: \"kubernetes.io/projected/1470c5b2-d2ad-4590-b08a-a41397b2b7b4-kube-api-access-swcj5\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:04.717263 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.717226    5838 generic.go:356] "Generic (PLEG): container finished" podID="1470c5b2-d2ad-4590-b08a-a41397b2b7b4" containerID="2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c" exitCode=0
Nov 03 18:22:04.717771 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.717277    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" event={"ID":"1470c5b2-d2ad-4590-b08a-a41397b2b7b4","Type":"ContainerDied","Data":"2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c"}
Nov 03 18:22:04.717771 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.717285    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"
Nov 03 18:22:04.717771 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.717318    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn" event={"ID":"1470c5b2-d2ad-4590-b08a-a41397b2b7b4","Type":"ContainerDied","Data":"293a4553e568ad6b0fa55a7ad83a070a088b2a364f44445560dbc9fa7be77eae"}
Nov 03 18:22:04.717771 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.717339    5838 scope.go:122] "RemoveContainer" containerID="2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c"
Nov 03 18:22:04.727300 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.727278    5838 scope.go:122] "RemoveContainer" containerID="2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c"
Nov 03 18:22:04.727621 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:22:04.727595    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c\": container with ID starting with 2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c not found: ID does not exist" containerID="2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c"
Nov 03 18:22:04.727676 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.727630    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c"} err="failed to get container status \"2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c\": rpc error: code = NotFound desc = could not find container \"2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c\": container with ID starting with 2e8a957a183d741be649b5ddec915d7f269bd0dfca665954489bf68807be5f7c not found: ID does not exist"
Nov 03 18:22:04.886511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.886473    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"]
Nov 03 18:22:04.937006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:04.936956    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adf17-6ccfb8df58-pjkpn"]
Nov 03 18:22:05.261339 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:05.261307    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:05.488650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:05.488610    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:05.647613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:05.647585    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:06.116178 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:06.116093    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1470c5b2-d2ad-4590-b08a-a41397b2b7b4" path="/var/lib/kubelet/pods/1470c5b2-d2ad-4590-b08a-a41397b2b7b4/volumes"
Nov 03 18:22:06.253098 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:06.253066    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:08.184064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:08.184025    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:08.184613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:08.184115    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:09.165133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:09.165097    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:22:10.339867 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:22:10.339818    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:22:10.717768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:10.717713    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:13.404631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:13.404594    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:13.561576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:13.561534    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:13.790075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:13.789971    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:14.304109 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:14.304067    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:15.864199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:15.864146    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:18.183803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:18.183758    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:18.184219 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:18.183760    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:19.403369 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.403332    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:19.816706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.816671    5838 generic.go:356] "Generic (PLEG): container finished" podID="c6a365a1-5679-46ce-9e4e-31421d540860" containerID="2421720b4c818d5a14b89348d9678f062a08e86af9639388167426e21ce3ab4e" exitCode=0
Nov 03 18:22:19.816904 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.816835    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" event={"ID":"c6a365a1-5679-46ce-9e4e-31421d540860","Type":"ContainerDied","Data":"2421720b4c818d5a14b89348d9678f062a08e86af9639388167426e21ce3ab4e"}
Nov 03 18:22:19.897596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.897287    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:22:19.993061 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.992993    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume\") pod \"c6a365a1-5679-46ce-9e4e-31421d540860\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") "
Nov 03 18:22:19.993212 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.993084    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp\" (UniqueName: \"kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp\") pod \"c6a365a1-5679-46ce-9e4e-31421d540860\" (UID: \"c6a365a1-5679-46ce-9e4e-31421d540860\") "
Nov 03 18:22:19.993396 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.993376    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume" pod "c6a365a1-5679-46ce-9e4e-31421d540860" (UID: "c6a365a1-5679-46ce-9e4e-31421d540860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:19.993556 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.993541    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a365a1-5679-46ce-9e4e-31421d540860-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:19.995712 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:19.995688    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp" pod "c6a365a1-5679-46ce-9e4e-31421d540860" (UID: "c6a365a1-5679-46ce-9e4e-31421d540860"). InnerVolumeSpecName "kube-api-access-gw2vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:20.005112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.005086    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:22:20.094839 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.094797    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78\" (UniqueName: \"kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78\") pod \"a73e1e65-3dcf-490d-987f-2510b787e372\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") "
Nov 03 18:22:20.095038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.094852    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume\") pod \"a73e1e65-3dcf-490d-987f-2510b787e372\" (UID: \"a73e1e65-3dcf-490d-987f-2510b787e372\") "
Nov 03 18:22:20.095167 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.095151    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2vp\" (UniqueName: \"kubernetes.io/projected/c6a365a1-5679-46ce-9e4e-31421d540860-kube-api-access-gw2vp\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:20.095312 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.095285    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume" pod "a73e1e65-3dcf-490d-987f-2510b787e372" (UID: "a73e1e65-3dcf-490d-987f-2510b787e372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:20.097105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.097079    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78" pod "a73e1e65-3dcf-490d-987f-2510b787e372" (UID: "a73e1e65-3dcf-490d-987f-2510b787e372"). InnerVolumeSpecName "kube-api-access-48t78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:20.196129 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.196100    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48t78\" (UniqueName: \"kubernetes.io/projected/a73e1e65-3dcf-490d-987f-2510b787e372-kube-api-access-48t78\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:20.196129 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.196126    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e1e65-3dcf-490d-987f-2510b787e372-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:20.824406 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.824372    5838 generic.go:356] "Generic (PLEG): container finished" podID="a73e1e65-3dcf-490d-987f-2510b787e372" containerID="17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d" exitCode=0
Nov 03 18:22:20.824951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.824418    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"
Nov 03 18:22:20.824951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.824453    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" event={"ID":"a73e1e65-3dcf-490d-987f-2510b787e372","Type":"ContainerDied","Data":"17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d"}
Nov 03 18:22:20.824951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.824506    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr" event={"ID":"a73e1e65-3dcf-490d-987f-2510b787e372","Type":"ContainerDied","Data":"703f352d53a2c1b5b28d361b2cbb841fee3432d8c869e70dcf1e1f22aba4cac2"}
Nov 03 18:22:20.824951 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.824528    5838 scope.go:122] "RemoveContainer" containerID="17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d"
Nov 03 18:22:20.826181 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.825933    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c" event={"ID":"c6a365a1-5679-46ce-9e4e-31421d540860","Type":"ContainerDied","Data":"ae02ea7d34ce8bde305f9b39c38990b171acc8582296fb5d282fe47d3290e961"}
Nov 03 18:22:20.826181 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.825963    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"
Nov 03 18:22:20.834648 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.834621    5838 scope.go:122] "RemoveContainer" containerID="17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d"
Nov 03 18:22:20.834992 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:22:20.834970    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d\": container with ID starting with 17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d not found: ID does not exist" containerID="17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d"
Nov 03 18:22:20.835087 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.835005    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d"} err="failed to get container status \"17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d\": rpc error: code = NotFound desc = could not find container \"17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d\": container with ID starting with 17482eb65e430fab17d41ac6759cac698058f7e893fa83d6c2d06e7d3366967d not found: ID does not exist"
Nov 03 18:22:20.835087 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.835031    5838 scope.go:122] "RemoveContainer" containerID="2421720b4c818d5a14b89348d9678f062a08e86af9639388167426e21ce3ab4e"
Nov 03 18:22:20.993331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:20.993244    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"]
Nov 03 18:22:21.044170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:21.044128    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adf17-85dd9f978f-62g7c"]
Nov 03 18:22:21.094117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:21.094088    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:21.113258 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:21.113225    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"]
Nov 03 18:22:21.119548 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:21.119518    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0adf17-d657b7b99-9wmrr"]
Nov 03 18:22:22.115662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:22.115623    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" path="/var/lib/kubelet/pods/a73e1e65-3dcf-490d-987f-2510b787e372/volumes"
Nov 03 18:22:22.116098 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:22.116028    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" path="/var/lib/kubelet/pods/c6a365a1-5679-46ce-9e4e-31421d540860/volumes"
Nov 03 18:22:26.758725 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:26.758687    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:26.913955 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:26.913917    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:40.339935 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:22:40.339890    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:22:40.552769 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:40.552685    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:42.104766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:42.104717    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:42.392772 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:42.392712    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:42.759071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:42.758975    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:22:43.088471 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:43.088376    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:47.469017 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:47.468979    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:49.949432 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:49.949396    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"]
Nov 03 18:22:49.949903 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:49.949638    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy" containerID="cri-o://564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680" gracePeriod=30
Nov 03 18:22:51.105062 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:51.105017    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:51.259650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:51.259615    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:51.485042 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:51.485013    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:55.203999 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.203970    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 18:22:55.264187 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264093    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264187 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264152    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264204    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264275    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264314    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264346    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264448    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264481    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:55.264665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264569    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264610    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:55.264665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264625    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert\") pod \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\" (UID: \"31406d2d-f93b-4f1b-bbf9-2e4aea874e94\") "
Nov 03 18:22:55.264665 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264629    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:55.265018 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.264702    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:55.265082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.265035    5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.265082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.265058    5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-workload-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.265082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.265073    5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-data\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.265416 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.265087    5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-credential-socket\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.265416 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.265199    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:55.266707 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.266682    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:55.266707 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.266692    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "kube-api-access-gmhjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:55.266910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.266794    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:55.266910 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.266810    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo" pod "31406d2d-f93b-4f1b-bbf9-2e4aea874e94" (UID: "31406d2d-f93b-4f1b-bbf9-2e4aea874e94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:22:55.366476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.366438    5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-podinfo\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.366476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.366472    5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-token\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.366476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.366483    5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istiod-ca-cert\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.366760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.366493    5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-istio-envoy\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:55.366760 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:55.366503    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhjf\" (UniqueName: \"kubernetes.io/projected/31406d2d-f93b-4f1b-bbf9-2e4aea874e94-kube-api-access-gmhjf\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.058825 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.058786    5838 generic.go:356] "Generic (PLEG): container finished" podID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerID="564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680" exitCode=0
Nov 03 18:22:56.059080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.058822    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" event={"ID":"31406d2d-f93b-4f1b-bbf9-2e4aea874e94","Type":"ContainerDied","Data":"564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680"}
Nov 03 18:22:56.059080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.058843    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"
Nov 03 18:22:56.059080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.058884    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk" event={"ID":"31406d2d-f93b-4f1b-bbf9-2e4aea874e94","Type":"ContainerDied","Data":"e3270902cb1ddadea0183bfa1f0596c75fbe5cf6c695db3fcebb49c74de822f6"}
Nov 03 18:22:56.059080 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.058907    5838 scope.go:122] "RemoveContainer" containerID="564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680"
Nov 03 18:22:56.069541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.069382    5838 scope.go:122] "RemoveContainer" containerID="564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680"
Nov 03 18:22:56.069711 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:22:56.069691    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680\": container with ID starting with 564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680 not found: ID does not exist" containerID="564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680"
Nov 03 18:22:56.069796 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.069720    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680"} err="failed to get container status \"564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680\": rpc error: code = NotFound desc = could not find container \"564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680\": container with ID starting with 564fa85c1b3b1c3f40b4d88771392d15a81235a947461ef614047e00a3ee1680 not found: ID does not exist"
Nov 03 18:22:56.257646 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.257611    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"]
Nov 03 18:22:56.307838 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.307804    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-82sdk"]
Nov 03 18:22:56.790190 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.790154    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:56.947712 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:56.947675    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:58.117587 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:58.117550    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" path="/var/lib/kubelet/pods/31406d2d-f93b-4f1b-bbf9-2e4aea874e94/volumes"
Nov 03 18:22:58.524862 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:58.524821    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:22:59.423922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:22:59.423882    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:06.456442 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:06.456402    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:10.338750 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:23:10.338696    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:23:17.859469 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:17.859433    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:25.928585 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:25.928552    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:33.178238 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:33.178198    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:35.347980 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.347946    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"]
Nov 03 18:23:35.348437 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348423    5838 cpu_manager.go:414] "containerMap: removing container" podUID="08da3b48-fdc8-4487-a7ce-7990cdec7c6a" containerName="revoke-keys"
Nov 03 18:23:35.348487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348438    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08da3b48-fdc8-4487-a7ce-7990cdec7c6a" containerName="revoke-keys"
Nov 03 18:23:35.348487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348462    5838 cpu_manager.go:414] "containerMap: removing container" podUID="1470c5b2-d2ad-4590-b08a-a41397b2b7b4" containerName="maas-api"
Nov 03 18:23:35.348487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348467    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1470c5b2-d2ad-4590-b08a-a41397b2b7b4" containerName="maas-api"
Nov 03 18:23:35.348487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348480    5838 cpu_manager.go:414] "containerMap: removing container" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" containerName="payload-pre-processing"
Nov 03 18:23:35.348487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348486    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" containerName="payload-pre-processing"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348503    5838 cpu_manager.go:414] "containerMap: removing container" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348508    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348520    5838 cpu_manager.go:414] "containerMap: removing container" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" containerName="payload-processing"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348525    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" containerName="payload-processing"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348533    5838 cpu_manager.go:414] "containerMap: removing container" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348539    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348602    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d79d5ae-44d1-4160-ae68-3e42df3a51a9" containerName="istio-proxy"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348608    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="08da3b48-fdc8-4487-a7ce-7990cdec7c6a" containerName="revoke-keys"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348617    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1470c5b2-d2ad-4590-b08a-a41397b2b7b4" containerName="maas-api"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348623    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73e1e65-3dcf-490d-987f-2510b787e372" containerName="payload-pre-processing"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348631    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="31406d2d-f93b-4f1b-bbf9-2e4aea874e94" containerName="istio-proxy"
Nov 03 18:23:35.348647 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.348637    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a365a1-5679-46ce-9e4e-31421d540860" containerName="payload-processing"
Nov 03 18:23:35.352976 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.352956    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.417962 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.417931    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.417971    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.417999    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.418074    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.418172    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvkv\" (UniqueName: \"kubernetes.io/projected/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-kube-api-access-wjvkv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.418219    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.418255    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418331 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.418285    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.418490 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.418366    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.460199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.460163    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"]
Nov 03 18:23:35.519148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519111    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvkv\" (UniqueName: \"kubernetes.io/projected/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-kube-api-access-wjvkv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519175    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519196    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519212    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519251    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519289    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519317    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519364 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519363    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519381    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519691    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.519777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519724    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.520012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519895    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.520012 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.519975    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.520291 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.520266    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.521823 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.521801    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.521905 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.521822    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.582762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.582714    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.582917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.582825    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvkv\" (UniqueName: \"kubernetes.io/projected/1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0-kube-api-access-wjvkv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jncwc\" (UID: \"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.667094 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.667060    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:35.975880 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:35.975777    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:36.102254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.102224    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"]
Nov 03 18:23:36.102443 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:23:36.102420    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1647bd7a_6878_40ad_b6b2_c4e06dcd9cc0.slice/crio-b4473eeec1ddaaf3da72ebd3ff181839c50d10c5acfb9e7eb852ac687d0c09b9 WatchSource:0}: Error finding container b4473eeec1ddaaf3da72ebd3ff181839c50d10c5acfb9e7eb852ac687d0c09b9: Status 404 returned error can't find the container with id b4473eeec1ddaaf3da72ebd3ff181839c50d10c5acfb9e7eb852ac687d0c09b9
Nov 03 18:23:36.104809 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.104776    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:36.104898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.104839    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:36.104898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.104871    5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:36.317477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.317386    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" event={"ID":"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0","Type":"ContainerStarted","Data":"758942b081f2ca9935220aea0c6e0cb840493f0c356aa0609f563b1dfe939790"}
Nov 03 18:23:36.317477 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.317444    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" event={"ID":"1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0","Type":"ContainerStarted","Data":"b4473eeec1ddaaf3da72ebd3ff181839c50d10c5acfb9e7eb852ac687d0c09b9"}
Nov 03 18:23:36.443637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.443576    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podStartSLOduration=1.443560838 podStartE2EDuration="1.443560838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:36.440943932 +0000 UTC m=+2952.451682033" watchObservedRunningTime="2026-11-03 18:23:36.443560838 +0000 UTC m=+2952.454298904"
Nov 03 18:23:36.667746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:36.667687    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:37.669872 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:37.669833    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:37.670290 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:37.669890    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:38.668202 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:38.668166    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:38.668402 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:38.668224    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:39.668103 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:39.668061    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:39.668608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:39.668128    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:40.348681 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:23:40.348644    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:23:40.668173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:40.668141    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:40.668591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:40.668195    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:41.668465 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:41.668424    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:41.669026 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:41.668490    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:42.667601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:42.667565    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:42.667857 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:42.667619    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:43.667782 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:43.667713    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jncwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:23:43.668341 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:43.667801    5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc" podUID="1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 18:23:44.672181 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:44.672150    5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:44.672582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:44.672448    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:44.673166 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:44.673150    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jncwc"
Nov 03 18:23:47.813723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:47.813669    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:48.889856 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:23:48.889824    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:10.338519 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:24:10.338432    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:24:12.631127 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:12.631090    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:24.123644 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:24.123613    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:24:24.759912 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:24.759877    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:24:24.770170 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:24.770144    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:24:40.338469 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:24:40.338433    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:24:45.519000 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:45.518961    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:24:45.753878 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:45.753815    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:45.909287 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:45.909249    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:48.156563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:48.156527    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:48.322664 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:48.322632    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:56.721481 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:24:56.721445    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:25:03.084944 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:03.084908    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:03.240236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:03.240195    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:10.339406 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:25:10.339363    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:25:20.655997 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:20.655952    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:25:28.900356 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:28.900301    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:34.014815 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:34.014685    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:25:35.563981 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:35.563927    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:35.726798 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:35.726717    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:40.338781 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:25:40.338712    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:25:48.509460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:25:48.509422    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:04.922220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:04.922183    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:26:07.084889 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:07.084843    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"]
Nov 03 18:26:07.154874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:07.154822    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0125bb-vdklv"]
Nov 03 18:26:07.373986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:07.373844    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:26:07.374339 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:07.374292    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:26:08.115999 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:08.115954    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08da3b48-fdc8-4487-a7ce-7990cdec7c6a" path="/var/lib/kubelet/pods/08da3b48-fdc8-4487-a7ce-7990cdec7c6a/volumes"
Nov 03 18:26:10.338351 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:26:10.338309    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:26:15.660131 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:15.660093    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:26:28.716501 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:28.716466    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:26:40.342017 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:26:40.341971    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:26:48.848457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:48.848411    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:53.816777 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:26:53.816739    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:27:03.934377 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:27:03.933245    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:27:10.342321 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:27:10.342271    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:27:25.253916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:27:25.253873    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:27:25.556704 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:27:25.556618    5838 scope.go:122] "RemoveContainer" containerID="787ae5cb66fc21a69faf6312f054e1ad5c29f3ad249f0c7f88241873b71f4471"
Nov 03 18:27:38.261812 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:27:38.261773    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:27:40.338586 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:27:40.338548    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:27:49.203892 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:27:49.203851    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:49.360922 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:27:49.360887    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:01.037142 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:01.037093    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:28:10.339017 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:28:10.338975    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:28:23.051933 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:23.051887    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:28:36.246597 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:36.246512    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:28:38.939758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:38.939184    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:28:38.939758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:38.939278    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:28:40.338784 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:28:40.338747    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:28:45.718686 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:45.718643    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:28:51.332884 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:28:51.332848    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:29:07.042930 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:07.042890    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:29:10.339658 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:29:10.339614    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:29:17.564901 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:17.564864    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:29:24.803897 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:24.803862    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:29:24.815392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:24.815365    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:29:34.128256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:34.128219    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:29:40.338861 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:29:40.338823    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:29:49.938201 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:49.938167    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:29:55.748637 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:29:55.748596    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:30:06.548305 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:06.548209    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:30:10.339582 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:30:10.339538    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:30:25.624121 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:25.624087    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:30:36.455189 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:36.455153    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:30:40.340040 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:30:40.339999    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:30:42.825778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:42.825720    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:43.510778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:43.510713    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:43.723959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:43.723919    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:47.821038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:47.820992    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:47.996336 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:47.996293    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:48.159264 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:48.159229    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:50.230958 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:50.230922    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:50.386007 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:50.385971    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:51.116708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:51.116671    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:51.299148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:51.299114    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:54.152072 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.152037    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"]
Nov 03 18:30:54.155915 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.155892    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.210650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.210619    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ca6dbf-serving-cert\""
Nov 03 18:30:54.224894 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.224855    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.225066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.224921    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r884\" (UniqueName: \"kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.225115 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.225083    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.225234 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.225215    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.262299 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.262261    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"]
Nov 03 18:30:54.326594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.326559    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r884\" (UniqueName: \"kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.326803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.326607    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.326803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.326670    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.326803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.326746    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.326933 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:30:54.326864    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-serving-cert: secret "maas-api-e2e-worker-single-main-ca6dbf-serving-cert" not found
Nov 03 18:30:54.326968 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:30:54.326935    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls podName:4f1e1d30-b285-45ab-ac6c-04d194e3358e nodeName:}" failed. No retries permitted until 2026-11-03 18:30:54.82691169 +0000 UTC m=+3390.837649757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls") pod "maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" (UID: "4f1e1d30-b285-45ab-ac6c-04d194e3358e") : secret "maas-api-e2e-worker-single-main-ca6dbf-serving-cert" not found
Nov 03 18:30:54.327493 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.327468    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.329479 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.329455    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.391556 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.391518    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r884\" (UniqueName: \"kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.395826 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.395793    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:54.556141 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.556027    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:54.831983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.831891    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:54.834529 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:54.834494    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:55.069941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:55.069891    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:55.451085 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:55.451053    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"]
Nov 03 18:30:55.453113 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:30:55.453072    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1e1d30_b285_45ab_ac6c_04d194e3358e.slice/crio-7c99a8bdf6d6ffdf8629e2738ea633761a0256aae5224069ae1dd92fe2139a9f WatchSource:0}: Error finding container 7c99a8bdf6d6ffdf8629e2738ea633761a0256aae5224069ae1dd92fe2139a9f: Status 404 returned error can't find the container with id 7c99a8bdf6d6ffdf8629e2738ea633761a0256aae5224069ae1dd92fe2139a9f
Nov 03 18:30:55.454511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:55.454486    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:30:56.432874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:56.432847    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" event={"ID":"4f1e1d30-b285-45ab-ac6c-04d194e3358e","Type":"ContainerStarted","Data":"7c99a8bdf6d6ffdf8629e2738ea633761a0256aae5224069ae1dd92fe2139a9f"}
Nov 03 18:30:57.440604 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:57.440562    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" event={"ID":"4f1e1d30-b285-45ab-ac6c-04d194e3358e","Type":"ContainerStarted","Data":"34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043"}
Nov 03 18:30:57.441254 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:57.440669    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:30:57.614073 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:57.614018    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" podStartSLOduration=2.709927457 podStartE2EDuration="3.614003212s" totalImagesPullingTime="904.075755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:57.612978606 +0000 UTC m=+3393.623716679" watchObservedRunningTime="2026-11-03 18:30:57.614003212 +0000 UTC m=+3393.624741283"
Nov 03 18:30:59.603591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:30:59.603552    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:02.651020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:02.650978    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:31:03.452943 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:03.452914    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:31:04.719895 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:04.719860    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:09.242709 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:09.242669    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:09.807534 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:09.807489    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:10.340337 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:31:10.340285    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:31:12.652256 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:12.652219    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:13.292794 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:13.292762    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:13.452397 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:13.452365    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:13.614134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:13.614044    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:15.214970 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:15.214934    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:16.135408 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:16.135378    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:20.285670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:20.285610    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:20.942576 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:20.942532    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:31:25.394996 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:25.394955    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:30.387253 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:30.387220    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:35.484064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:35.483980    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:40.339646 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:31:40.339602    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:31:40.493517 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:40.493480    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:45.584799 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:45.584761    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:47.132116 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:47.132078    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:31:50.573376 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:50.573345    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:50.727954 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:50.727913    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:55.644670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:31:55.644614    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:00.689650 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:00.689612    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:05.786190 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:05.786144    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:10.339034 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:32:10.338992    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:32:10.803720 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:10.803680    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:12.983232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:12.983177    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:16.103543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:16.103506    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:16.351982 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:16.351947    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:32:18.045357 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:18.045321    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:21.395062 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:21.395029    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:25.438099 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:25.438059    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:25.589941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:25.589902    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:25.731395 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:25.731277    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"]
Nov 03 18:32:25.731619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:25.731550    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" podUID="4f1e1d30-b285-45ab-ac6c-04d194e3358e" containerName="maas-api" containerID="cri-o://34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043" gracePeriod=30
Nov 03 18:32:25.920124 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:25.920078    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:26.993766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:26.993715    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:32:27.010236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.010201    5838 generic.go:356] "Generic (PLEG): container finished" podID="4f1e1d30-b285-45ab-ac6c-04d194e3358e" containerID="34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043" exitCode=0
Nov 03 18:32:27.010425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.010253    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"
Nov 03 18:32:27.010425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.010288    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" event={"ID":"4f1e1d30-b285-45ab-ac6c-04d194e3358e","Type":"ContainerDied","Data":"34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043"}
Nov 03 18:32:27.010425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.010347    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt" event={"ID":"4f1e1d30-b285-45ab-ac6c-04d194e3358e","Type":"ContainerDied","Data":"7c99a8bdf6d6ffdf8629e2738ea633761a0256aae5224069ae1dd92fe2139a9f"}
Nov 03 18:32:27.010425 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.010374    5838 scope.go:122] "RemoveContainer" containerID="34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043"
Nov 03 18:32:27.020016 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.019992    5838 scope.go:122] "RemoveContainer" containerID="34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043"
Nov 03 18:32:27.020314 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:32:27.020287    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043\": container with ID starting with 34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043 not found: ID does not exist" containerID="34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043"
Nov 03 18:32:27.020382 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.020325    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043"} err="failed to get container status \"34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043\": rpc error: code = NotFound desc = could not find container \"34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043\": container with ID starting with 34f1dc57c9899522cd41ec2dad7776853241ef17f35e4afd822e5bad53393043 not found: ID does not exist"
Nov 03 18:32:27.127746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.127619    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884\" (UniqueName: \"kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884\") pod \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") "
Nov 03 18:32:27.127971 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.127819    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\") pod \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") "
Nov 03 18:32:27.127971 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.127869    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs\") pod \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") "
Nov 03 18:32:27.127971 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.127908    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca\") pod \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\" (UID: \"4f1e1d30-b285-45ab-ac6c-04d194e3358e\") "
Nov 03 18:32:27.128548 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.128519    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca" pod "4f1e1d30-b285-45ab-ac6c-04d194e3358e" (UID: "4f1e1d30-b285-45ab-ac6c-04d194e3358e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:32:27.130454 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.130415    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls" pod "4f1e1d30-b285-45ab-ac6c-04d194e3358e" (UID: "4f1e1d30-b285-45ab-ac6c-04d194e3358e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:32:27.130839 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.130806    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884" pod "4f1e1d30-b285-45ab-ac6c-04d194e3358e" (UID: "4f1e1d30-b285-45ab-ac6c-04d194e3358e"). InnerVolumeSpecName "kube-api-access-8r884". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:32:27.131094 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.131071    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs" pod "4f1e1d30-b285-45ab-ac6c-04d194e3358e" (UID: "4f1e1d30-b285-45ab-ac6c-04d194e3358e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:32:27.229514 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.229478    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:27.229514 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.229514    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1e1d30-b285-45ab-ac6c-04d194e3358e-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:27.229822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.229528    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1e1d30-b285-45ab-ac6c-04d194e3358e-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:27.229822 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.229538    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r884\" (UniqueName: \"kubernetes.io/projected/4f1e1d30-b285-45ab-ac6c-04d194e3358e-kube-api-access-8r884\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:27.442225 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.442190    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"]
Nov 03 18:32:27.492514 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:27.492481    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca6dbf-6999f684fb-69gvt"]
Nov 03 18:32:28.116079 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:28.116044    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1e1d30-b285-45ab-ac6c-04d194e3358e" path="/var/lib/kubelet/pods/4f1e1d30-b285-45ab-ac6c-04d194e3358e/volumes"
Nov 03 18:32:29.612387 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:29.612353    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:29.910985 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:29.910952    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:30.706450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:30.706411    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:30.830025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:30.829971    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:32:37.803528 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:37.803475    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:37.959880 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:37.959844    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:38.187020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:38.186982    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:40.338210 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:32:40.338171    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:32:50.928946 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:50.928907    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:51.842049 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:51.842011    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:54.638849 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:54.638814    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:32:55.787755 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:55.787476    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:56.860999 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:32:56.860966    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:03.500595 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:03.500502    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:05.630522 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:05.630483    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:33:10.338629 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:33:10.338585    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:33:14.451985 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.451939    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"]
Nov 03 18:33:14.452515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.452212    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr" podUID="ac802f81-9b7f-4abe-9b66-be17114d4a7c" containerName="manager" containerID="cri-o://9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03" gracePeriod=10
Nov 03 18:33:14.722849 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.722818    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 18:33:14.777582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.777546    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume\") pod \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") "
Nov 03 18:33:14.777800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.777656    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8\" (UniqueName: \"kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8\") pod \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\" (UID: \"ac802f81-9b7f-4abe-9b66-be17114d4a7c\") "
Nov 03 18:33:14.778039 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.778015    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume" pod "ac802f81-9b7f-4abe-9b66-be17114d4a7c" (UID: "ac802f81-9b7f-4abe-9b66-be17114d4a7c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:14.780058 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.780037    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8" pod "ac802f81-9b7f-4abe-9b66-be17114d4a7c" (UID: "ac802f81-9b7f-4abe-9b66-be17114d4a7c"). InnerVolumeSpecName "kube-api-access-gd4s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:14.878785 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.878744    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4s8\" (UniqueName: \"kubernetes.io/projected/ac802f81-9b7f-4abe-9b66-be17114d4a7c-kube-api-access-gd4s8\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:33:14.878785 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:14.878780    5838 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac802f81-9b7f-4abe-9b66-be17114d4a7c-extensions-socket-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:33:15.340359 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.340317    5838 generic.go:356] "Generic (PLEG): container finished" podID="ac802f81-9b7f-4abe-9b66-be17114d4a7c" containerID="9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03" exitCode=0
Nov 03 18:33:15.340544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.340366    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"
Nov 03 18:33:15.340544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.340386    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr" event={"ID":"ac802f81-9b7f-4abe-9b66-be17114d4a7c","Type":"ContainerDied","Data":"9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03"}
Nov 03 18:33:15.340544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.340439    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr" event={"ID":"ac802f81-9b7f-4abe-9b66-be17114d4a7c","Type":"ContainerDied","Data":"24e976d70a507d93e6c94a070b3d54b88886aa0f9cd3f2b6b7066e44bd61eeb7"}
Nov 03 18:33:15.340544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.340468    5838 scope.go:122] "RemoveContainer" containerID="9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03"
Nov 03 18:33:15.352005 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.351982    5838 scope.go:122] "RemoveContainer" containerID="9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03"
Nov 03 18:33:15.352291 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:33:15.352271    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03\": container with ID starting with 9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03 not found: ID does not exist" containerID="9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03"
Nov 03 18:33:15.352348 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.352309    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03"} err="failed to get container status \"9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03\": rpc error: code = NotFound desc = could not find container \"9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03\": container with ID starting with 9a0e1cb498c73f9523a170756a0e97d38160565305d464ffbf9c68ffc339ba03 not found: ID does not exist"
Nov 03 18:33:15.510184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.510150    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"]
Nov 03 18:33:15.565615 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:15.565574    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v9vxr"]
Nov 03 18:33:16.115412 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:16.115379    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac802f81-9b7f-4abe-9b66-be17114d4a7c" path="/var/lib/kubelet/pods/ac802f81-9b7f-4abe-9b66-be17114d4a7c/volumes"
Nov 03 18:33:22.256340 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:22.256292    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:40.338428 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:33:40.338385    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:33:51.267968 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:33:51.267920    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:10.338648 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:34:10.338605    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:34:18.073649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.073608    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"]
Nov 03 18:34:18.074366 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.074342    5838 cpu_manager.go:414] "containerMap: removing container" podUID="ac802f81-9b7f-4abe-9b66-be17114d4a7c" containerName="manager"
Nov 03 18:34:18.074460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.074370    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac802f81-9b7f-4abe-9b66-be17114d4a7c" containerName="manager"
Nov 03 18:34:18.074460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.074406    5838 cpu_manager.go:414] "containerMap: removing container" podUID="4f1e1d30-b285-45ab-ac6c-04d194e3358e" containerName="maas-api"
Nov 03 18:34:18.074460 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.074415    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1e1d30-b285-45ab-ac6c-04d194e3358e" containerName="maas-api"
Nov 03 18:34:18.074610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.074556    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1e1d30-b285-45ab-ac6c-04d194e3358e" containerName="maas-api"
Nov 03 18:34:18.074610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.074581    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac802f81-9b7f-4abe-9b66-be17114d4a7c" containerName="manager"
Nov 03 18:34:18.078025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.078001    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.131876 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.131842    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"]
Nov 03 18:34:18.139002 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.138970    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67d70366-124c-4a0a-ac40-76957d0331e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-c8sr8\" (UID: \"67d70366-124c-4a0a-ac40-76957d0331e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.139159 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.139120    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lf48\" (UniqueName: \"kubernetes.io/projected/67d70366-124c-4a0a-ac40-76957d0331e5-kube-api-access-2lf48\") pod \"kuadrant-operator-controller-manager-678d87766f-c8sr8\" (UID: \"67d70366-124c-4a0a-ac40-76957d0331e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.179548 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.179510    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-snz9x\""
Nov 03 18:34:18.240566 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.240525    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lf48\" (UniqueName: \"kubernetes.io/projected/67d70366-124c-4a0a-ac40-76957d0331e5-kube-api-access-2lf48\") pod \"kuadrant-operator-controller-manager-678d87766f-c8sr8\" (UID: \"67d70366-124c-4a0a-ac40-76957d0331e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.240780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.240626    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67d70366-124c-4a0a-ac40-76957d0331e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-c8sr8\" (UID: \"67d70366-124c-4a0a-ac40-76957d0331e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.241043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.241025    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67d70366-124c-4a0a-ac40-76957d0331e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-c8sr8\" (UID: \"67d70366-124c-4a0a-ac40-76957d0331e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.304455 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.304415    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lf48\" (UniqueName: \"kubernetes.io/projected/67d70366-124c-4a0a-ac40-76957d0331e5-kube-api-access-2lf48\") pod \"kuadrant-operator-controller-manager-678d87766f-c8sr8\" (UID: \"67d70366-124c-4a0a-ac40-76957d0331e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.390805 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.390770    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:18.826600 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:34:18.826562    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d70366_124c_4a0a_ac40_76957d0331e5.slice/crio-6d5bb7ff9aaee8a2265433827b7777278dac2611a74c60a54cef726dbefff4d3 WatchSource:0}: Error finding container 6d5bb7ff9aaee8a2265433827b7777278dac2611a74c60a54cef726dbefff4d3: Status 404 returned error can't find the container with id 6d5bb7ff9aaee8a2265433827b7777278dac2611a74c60a54cef726dbefff4d3
Nov 03 18:34:18.875360 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:18.875328    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"]
Nov 03 18:34:19.717965 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:19.717925    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8" event={"ID":"67d70366-124c-4a0a-ac40-76957d0331e5","Type":"ContainerStarted","Data":"bcb8cceae2cbc40bf3ca7f3a8f8f11c7d3656a7a3fbad0ffebe4e55889cada30"}
Nov 03 18:34:19.718383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:19.717980    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8" event={"ID":"67d70366-124c-4a0a-ac40-76957d0331e5","Type":"ContainerStarted","Data":"6d5bb7ff9aaee8a2265433827b7777278dac2611a74c60a54cef726dbefff4d3"}
Nov 03 18:34:19.718383 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:19.718034    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:19.943651 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:19.943591    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8" podStartSLOduration=2.943574986 podStartE2EDuration="2.943574986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:34:19.890792403 +0000 UTC m=+3595.901530494" watchObservedRunningTime="2026-11-03 18:34:19.943574986 +0000 UTC m=+3595.954313061"
Nov 03 18:34:24.847317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:24.847282    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:34:24.861087 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:24.861056    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:34:30.726523 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:30.726490    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-c8sr8"
Nov 03 18:34:40.340813 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:34:40.340762    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:34:40.831778 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:40.831719    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:34:51.666916 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:51.666877    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:58.439469 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:34:58.439429    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:35:08.155398 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:08.155357    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:35:10.338109 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:35:10.338072    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:35:27.146317 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:27.146277    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:27.536874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:27.536786    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:35:27.896067 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:27.896031    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:28.051659 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:28.051628    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:32.429282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:32.429251    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:32.601748 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:32.601706    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:32.768145 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:32.768042    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:35.734263 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:35.734225    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:35.914535 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:35.914500    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:38.837987 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.837946    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"]
Nov 03 18:35:38.841918 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.841896    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:38.896705 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.896674    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-057118-serving-cert\""
Nov 03 18:35:38.925338 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.925305    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gx8m\" (UniqueName: \"kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:38.925527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.925346    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:38.925527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.925364    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:38.925527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.925504    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:38.949631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:38.949595    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"]
Nov 03 18:35:39.026563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.026522    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.026563 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.026564    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.026834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.026667    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.026834 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:35:39.026818    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-serving-cert: secret "maas-api-e2e-worker-single-main-057118-serving-cert" not found
Nov 03 18:35:39.026908 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:35:39.026880    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls podName:1e3ce732-7849-4309-93f6-4e2dfcd0aef9 nodeName:}" failed. No retries permitted until 2026-11-03 18:35:39.526864007 +0000 UTC m=+3675.537602061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls") pod "maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" (UID: "1e3ce732-7849-4309-93f6-4e2dfcd0aef9") : secret "maas-api-e2e-worker-single-main-057118-serving-cert" not found
Nov 03 18:35:39.026908 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.026822    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gx8m\" (UniqueName: \"kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.027337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.027317    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.029313 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.029290    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.037863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.037833    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:39.097095 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.096870    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gx8m\" (UniqueName: \"kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.247896 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.247859    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:39.412591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.412543    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:39.532249 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.532211    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.534863 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.534834    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:39.760077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:39.759977    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:40.141452 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:40.141427    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"]
Nov 03 18:35:40.143312 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:35:40.143282    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3ce732_7849_4309_93f6_4e2dfcd0aef9.slice/crio-e442ba9730db6b5eae4f8332005674e1293f3749a632e6d34357ca0391ae91e9 WatchSource:0}: Error finding container e442ba9730db6b5eae4f8332005674e1293f3749a632e6d34357ca0391ae91e9: Status 404 returned error can't find the container with id e442ba9730db6b5eae4f8332005674e1293f3749a632e6d34357ca0391ae91e9
Nov 03 18:35:40.255773 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:40.255708    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" event={"ID":"1e3ce732-7849-4309-93f6-4e2dfcd0aef9","Type":"ContainerStarted","Data":"e442ba9730db6b5eae4f8332005674e1293f3749a632e6d34357ca0391ae91e9"}
Nov 03 18:35:40.339597 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:35:40.339560    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:35:41.263322 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:41.263281    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" event={"ID":"1e3ce732-7849-4309-93f6-4e2dfcd0aef9","Type":"ContainerStarted","Data":"0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476"}
Nov 03 18:35:41.263852 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:41.263395    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:41.386006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:41.385901    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" podStartSLOduration=2.501810259 podStartE2EDuration="3.385886687s" totalImagesPullingTime="884.076428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:41.383798274 +0000 UTC m=+3677.394536367" watchObservedRunningTime="2026-11-03 18:35:41.385886687 +0000 UTC m=+3677.396624756"
Nov 03 18:35:44.191886 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:44.191855    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:47.275306 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:47.275265    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:35:47.439537 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:47.439499    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:35:49.263828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:49.263791    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:51.958483 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:51.958444    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:54.305109 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:54.305064    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:57.214497 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:57.214462    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:57.816184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:57.816139    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:57.971376 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:57.971336    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:58.130182 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:58.130104    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:59.718297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:35:59.718261    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:02.136487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:02.136454    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:04.797780 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:04.797680    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:06.725512 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:06.725481    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:36:09.886025 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:09.885986    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:10.339122 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:36:10.339082    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:36:14.897594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:14.897559    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:19.999139 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:19.999094    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:25.009648 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:25.009610    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:30.181304 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:30.181263    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:31.113148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:31.113115    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:36:35.193706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:35.193667    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:40.194412 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:40.194378    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:40.338992 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:36:40.338949    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:36:45.284390 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:45.284359    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:50.296134 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:50.296099    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:52.303004 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:52.302962    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:55.289444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:55.289412    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:57.094614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:57.094573    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:59.016980 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:36:59.016944    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:37:00.684905 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:00.684867    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:02.165895 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:02.165850    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:05.819482 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:05.819449    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:09.524028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:09.523996    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:09.682464 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:09.682428    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:10.135270 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:10.135229    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"]
Nov 03 18:37:10.135464 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:10.135435    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" podUID="1e3ce732-7849-4309-93f6-4e2dfcd0aef9" containerName="maas-api" containerID="cri-o://0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476" gracePeriod=30
Nov 03 18:37:10.237202 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:10.237163    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:10.348544 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:37:10.348433    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:37:11.398105 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.398079    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:37:11.504803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.504706    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m\" (UniqueName: \"kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m\") pod \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") "
Nov 03 18:37:11.504992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.504899    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs\") pod \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") "
Nov 03 18:37:11.504992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.504932    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca\") pod \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") "
Nov 03 18:37:11.504992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.504981    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\") pod \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\" (UID: \"1e3ce732-7849-4309-93f6-4e2dfcd0aef9\") "
Nov 03 18:37:11.505337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.505309    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca" pod "1e3ce732-7849-4309-93f6-4e2dfcd0aef9" (UID: "1e3ce732-7849-4309-93f6-4e2dfcd0aef9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:11.507173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.507148    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m" pod "1e3ce732-7849-4309-93f6-4e2dfcd0aef9" (UID: "1e3ce732-7849-4309-93f6-4e2dfcd0aef9"). InnerVolumeSpecName "kube-api-access-7gx8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:11.507688 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.507662    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls" pod "1e3ce732-7849-4309-93f6-4e2dfcd0aef9" (UID: "1e3ce732-7849-4309-93f6-4e2dfcd0aef9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:11.507688 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.507678    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs" pod "1e3ce732-7849-4309-93f6-4e2dfcd0aef9" (UID: "1e3ce732-7849-4309-93f6-4e2dfcd0aef9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:11.607033 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.606992    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:11.607033 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.607027    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:11.607033 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.607037    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:11.607287 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.607046    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gx8m\" (UniqueName: \"kubernetes.io/projected/1e3ce732-7849-4309-93f6-4e2dfcd0aef9-kube-api-access-7gx8m\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:11.843184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.843021    5838 generic.go:356] "Generic (PLEG): container finished" podID="1e3ce732-7849-4309-93f6-4e2dfcd0aef9" containerID="0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476" exitCode=0
Nov 03 18:37:11.843431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.843191    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"
Nov 03 18:37:11.843431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.843399    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" event={"ID":"1e3ce732-7849-4309-93f6-4e2dfcd0aef9","Type":"ContainerDied","Data":"0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476"}
Nov 03 18:37:11.843568 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.843549    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv" event={"ID":"1e3ce732-7849-4309-93f6-4e2dfcd0aef9","Type":"ContainerDied","Data":"e442ba9730db6b5eae4f8332005674e1293f3749a632e6d34357ca0391ae91e9"}
Nov 03 18:37:11.843834 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.843819    5838 scope.go:122] "RemoveContainer" containerID="0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476"
Nov 03 18:37:11.862096 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.862069    5838 scope.go:122] "RemoveContainer" containerID="0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476"
Nov 03 18:37:11.862485 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:37:11.862455    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476\": container with ID starting with 0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476 not found: ID does not exist" containerID="0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476"
Nov 03 18:37:11.862550 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:11.862499    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476"} err="failed to get container status \"0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476\": rpc error: code = NotFound desc = could not find container \"0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476\": container with ID starting with 0753553a69b156528a167e11d77767747517e157c1232f7744afb482bc310476 not found: ID does not exist"
Nov 03 18:37:12.013407 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:12.013371    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"]
Nov 03 18:37:12.064043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:12.064009    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-057118-5d6cc9bd8d-2f9zv"]
Nov 03 18:37:12.115482 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:12.115392    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3ce732-7849-4309-93f6-4e2dfcd0aef9" path="/var/lib/kubelet/pods/1e3ce732-7849-4309-93f6-4e2dfcd0aef9/volumes"
Nov 03 18:37:13.884184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:13.884146    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:14.172354 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:14.172320    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:14.936155 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:14.936125    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:16.941568 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:16.941530    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:37:22.046228 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:22.046187    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:22.198840 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:22.198805    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:22.389862 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:22.389809    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:29.704565 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:29.704531    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:34.320034 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:34.319943    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:37:37.373938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:37.373893    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:37:37.374658 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:37.374621    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:37:39.120711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:39.120678    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:40.339785 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:37:40.339714    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:37:40.723071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:40.723033    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:52.647133 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:52.647090    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:57.346297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:37:57.346267    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:10.339177 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:38:10.339138    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:38:15.313165 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:15.313126    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:20.132965 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:20.132927    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:38:23.926947 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:23.926887    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-ml2hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:38:23.927378 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:23.927004    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ml2hv" podUID="efc862fa-db10-4720-9d9b-90d4533a216c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:38:40.339760 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:38:40.339702    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:38:47.560864 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:47.560825    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:38:52.997116 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:52.997074    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:59.054077 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:38:59.054033    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:03.124222 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:03.124140    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:39:07.688683 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:07.688644    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:10.338580 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:39:10.338536    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:39:17.826083 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:17.826047    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:22.340112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:22.340072    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:39:25.070571 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:25.070528    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:39:25.087711 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:25.087671    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:39:34.823776 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:34.823715    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:39:40.351305 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:39:40.351259    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:39:46.021803 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:46.021767    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:39:53.393421 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:39:53.393369    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:07.646097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:07.646042    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:40:08.136506 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.136417    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"]
Nov 03 18:40:08.136978 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.136962    5838 cpu_manager.go:414] "containerMap: removing container" podUID="1e3ce732-7849-4309-93f6-4e2dfcd0aef9" containerName="maas-api"
Nov 03 18:40:08.137044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.136981    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3ce732-7849-4309-93f6-4e2dfcd0aef9" containerName="maas-api"
Nov 03 18:40:08.137089 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.137078    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3ce732-7849-4309-93f6-4e2dfcd0aef9" containerName="maas-api"
Nov 03 18:40:08.151084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.151003    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.257871 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.257835    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"]
Nov 03 18:40:08.262038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.262009    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4d845-94ca-4ce6-847c-eeccad6ce534-metrics-certs\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.262190 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.262045    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4d845-94ca-4ce6-847c-eeccad6ce534-openshift-service-ca\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.262190 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.262073    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4d845-94ca-4ce6-847c-eeccad6ce534-maas-api-tls\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.262292 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.262272    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26b97\" (UniqueName: \"kubernetes.io/projected/c2b4d845-94ca-4ce6-847c-eeccad6ce534-kube-api-access-26b97\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.363380 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.363347    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26b97\" (UniqueName: \"kubernetes.io/projected/c2b4d845-94ca-4ce6-847c-eeccad6ce534-kube-api-access-26b97\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.363569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.363450    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4d845-94ca-4ce6-847c-eeccad6ce534-metrics-certs\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.363569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.363475    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4d845-94ca-4ce6-847c-eeccad6ce534-openshift-service-ca\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.363569 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.363497    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4d845-94ca-4ce6-847c-eeccad6ce534-maas-api-tls\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.364188 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.364165    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4d845-94ca-4ce6-847c-eeccad6ce534-openshift-service-ca\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.366374 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.366349    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4d845-94ca-4ce6-847c-eeccad6ce534-metrics-certs\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.366501 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.366386    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4d845-94ca-4ce6-847c-eeccad6ce534-maas-api-tls\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.427165 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.427137    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26b97\" (UniqueName: \"kubernetes.io/projected/c2b4d845-94ca-4ce6-847c-eeccad6ce534-kube-api-access-26b97\") pod \"maas-api-7456bf56b5-kmlfq\" (UID: \"c2b4d845-94ca-4ce6-847c-eeccad6ce534\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.467933 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.467899    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:08.851117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.851093    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"]
Nov 03 18:40:08.853519 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:40:08.853488    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b4d845_94ca_4ce6_847c_eeccad6ce534.slice/crio-c8970984de000d027fba1e85fa9a2a24e260c0940aecb8ce05a1ad2cf11052d9 WatchSource:0}: Error finding container c8970984de000d027fba1e85fa9a2a24e260c0940aecb8ce05a1ad2cf11052d9: Status 404 returned error can't find the container with id c8970984de000d027fba1e85fa9a2a24e260c0940aecb8ce05a1ad2cf11052d9
Nov 03 18:40:08.854961 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:08.854941    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:40:09.119599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:09.119506    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq" event={"ID":"c2b4d845-94ca-4ce6-847c-eeccad6ce534","Type":"ContainerStarted","Data":"c8970984de000d027fba1e85fa9a2a24e260c0940aecb8ce05a1ad2cf11052d9"}
Nov 03 18:40:10.127630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:10.127542    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq" event={"ID":"c2b4d845-94ca-4ce6-847c-eeccad6ce534","Type":"ContainerStarted","Data":"c86ec3fb743152a8a62e5fa8f72229c61106607f439edf3c1e413a5fde7e5698"}
Nov 03 18:40:10.127630 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:10.127602    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:10.251325 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:10.251270    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq" podStartSLOduration=1.43070001 podStartE2EDuration="2.251254989s" totalImagesPullingTime="820.554979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:10.248969593 +0000 UTC m=+3946.259707700" watchObservedRunningTime="2026-11-03 18:40:10.251254989 +0000 UTC m=+3946.261993062"
Nov 03 18:40:10.339452 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:40:10.339415    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:40:11.156123 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:11.156092    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:11.318584 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:11.318542    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:14.750445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:14.750409    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:15.379793 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:15.379756    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:15.547075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:15.547042    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:16.139615 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:16.139585    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-kmlfq"
Nov 03 18:40:16.349282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:16.349248    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"]
Nov 03 18:40:16.349489 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:16.349444    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" podUID="878c563c-ed19-4910-8aef-023edd6e9376" containerName="maas-api" containerID="cri-o://44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d" gracePeriod=30
Nov 03 18:40:17.094868 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.094817    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" podUID="878c563c-ed19-4910-8aef-023edd6e9376" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused"
Nov 03 18:40:17.619172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.619143    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 18:40:17.658474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.658429    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls\") pod \"878c563c-ed19-4910-8aef-023edd6e9376\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") "
Nov 03 18:40:17.658474 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.658477    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs\") pod \"878c563c-ed19-4910-8aef-023edd6e9376\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") "
Nov 03 18:40:17.658718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.658627    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx\" (UniqueName: \"kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx\") pod \"878c563c-ed19-4910-8aef-023edd6e9376\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") "
Nov 03 18:40:17.658815 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.658785    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca\") pod \"878c563c-ed19-4910-8aef-023edd6e9376\" (UID: \"878c563c-ed19-4910-8aef-023edd6e9376\") "
Nov 03 18:40:17.659117 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.659087    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca" pod "878c563c-ed19-4910-8aef-023edd6e9376" (UID: "878c563c-ed19-4910-8aef-023edd6e9376"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:17.661064 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.661041    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx" pod "878c563c-ed19-4910-8aef-023edd6e9376" (UID: "878c563c-ed19-4910-8aef-023edd6e9376"). InnerVolumeSpecName "kube-api-access-2cvlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:40:17.661337 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.661320    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls" pod "878c563c-ed19-4910-8aef-023edd6e9376" (UID: "878c563c-ed19-4910-8aef-023edd6e9376"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:17.661420 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.661341    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs" pod "878c563c-ed19-4910-8aef-023edd6e9376" (UID: "878c563c-ed19-4910-8aef-023edd6e9376"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:17.760038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.759950    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/878c563c-ed19-4910-8aef-023edd6e9376-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:17.760038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.759982    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:17.760038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.759991    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878c563c-ed19-4910-8aef-023edd6e9376-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:17.760038 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:17.760000    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvlx\" (UniqueName: \"kubernetes.io/projected/878c563c-ed19-4910-8aef-023edd6e9376-kube-api-access-2cvlx\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:18.178852 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.178812    5838 generic.go:356] "Generic (PLEG): container finished" podID="878c563c-ed19-4910-8aef-023edd6e9376" containerID="44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d" exitCode=0
Nov 03 18:40:18.179071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.178868    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"
Nov 03 18:40:18.179071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.178902    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" event={"ID":"878c563c-ed19-4910-8aef-023edd6e9376","Type":"ContainerDied","Data":"44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d"}
Nov 03 18:40:18.179071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.178967    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7" event={"ID":"878c563c-ed19-4910-8aef-023edd6e9376","Type":"ContainerDied","Data":"b5ff72575c8450c0a1f7b9164a3a02eeb838739481a436b4eee8d6a8500e1570"}
Nov 03 18:40:18.179071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.178990    5838 scope.go:122] "RemoveContainer" containerID="44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d"
Nov 03 18:40:18.188927 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.188908    5838 scope.go:122] "RemoveContainer" containerID="44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d"
Nov 03 18:40:18.189248 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:40:18.189224    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d\": container with ID starting with 44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d not found: ID does not exist" containerID="44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d"
Nov 03 18:40:18.189370 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.189254    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d"} err="failed to get container status \"44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d\": rpc error: code = NotFound desc = could not find container \"44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d\": container with ID starting with 44ec14e8f0a6fb69d0d408918941613d1edb93ea83ee90f1087d59bafbfdc93d not found: ID does not exist"
Nov 03 18:40:18.348629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.348590    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"]
Nov 03 18:40:18.354984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.354949    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-jkbj7"]
Nov 03 18:40:18.564901 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.564867    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:18.728674 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.728643    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:18.881265 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:18.881169    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:20.123655 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:20.123617    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878c563c-ed19-4910-8aef-023edd6e9376" path="/var/lib/kubelet/pods/878c563c-ed19-4910-8aef-023edd6e9376/volumes"
Nov 03 18:40:21.810574 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:21.810536    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:21.990572 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:21.990531    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:22.237718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.237681    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"]
Nov 03 18:40:22.238181 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.238168    5838 cpu_manager.go:414] "containerMap: removing container" podUID="878c563c-ed19-4910-8aef-023edd6e9376" containerName="maas-api"
Nov 03 18:40:22.238230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.238182    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878c563c-ed19-4910-8aef-023edd6e9376" containerName="maas-api"
Nov 03 18:40:22.238289 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.238281    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="878c563c-ed19-4910-8aef-023edd6e9376" containerName="maas-api"
Nov 03 18:40:22.242923 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.242903    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.301071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.301027    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"]
Nov 03 18:40:22.309972 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.309937    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.310163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.310040    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbl2\" (UniqueName: \"kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2\") pod \"payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.348086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.348050    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c188b2e5-dockercfg-dqvw8\""
Nov 03 18:40:22.348086 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.348064    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c188b2e5\""
Nov 03 18:40:22.394320 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.394286    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"]
Nov 03 18:40:22.398173 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.398156    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.411387 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.411358    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.411525 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.411420    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbl2\" (UniqueName: \"kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2\") pod \"payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.412040 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.412017    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.428375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.428340    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"]
Nov 03 18:40:22.474676 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.474641    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbl2\" (UniqueName: \"kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2\") pod \"payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.512813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.512693    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume\") pod \"payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") " pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.512813 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.512776    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjp8v\" (UniqueName: \"kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v\") pod \"payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") " pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.557607 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.557565    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:22.614814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.614601    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume\") pod \"payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") " pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.614814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.614669    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjp8v\" (UniqueName: \"kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v\") pod \"payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") " pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.618643 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.615396    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume\") pod \"payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") " pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.678163 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.678127    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjp8v\" (UniqueName: \"kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v\") pod \"payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") " pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.711799 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.711746    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:22.937503 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:22.937473    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"]
Nov 03 18:40:22.939381 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:40:22.939343    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d269d6_6bdb_4c38_b1eb_d0fd0ee6e3a2.slice/crio-aae0a92eb917a15e88798aa35fc10fcf086ffed48d8d424bad97212976bb3d9f WatchSource:0}: Error finding container aae0a92eb917a15e88798aa35fc10fcf086ffed48d8d424bad97212976bb3d9f: Status 404 returned error can't find the container with id aae0a92eb917a15e88798aa35fc10fcf086ffed48d8d424bad97212976bb3d9f
Nov 03 18:40:23.093007 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.092982    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"]
Nov 03 18:40:23.095257 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:40:23.095221    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866ab07a_6408_4046_8fb0_55b85e275a36.slice/crio-4a457d2896f1d0b3626c17a728956ffdeb3bc1317f30972bef574652eb57e430 WatchSource:0}: Error finding container 4a457d2896f1d0b3626c17a728956ffdeb3bc1317f30972bef574652eb57e430: Status 404 returned error can't find the container with id 4a457d2896f1d0b3626c17a728956ffdeb3bc1317f30972bef574652eb57e430
Nov 03 18:40:23.212247 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.212210    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d" event={"ID":"866ab07a-6408-4046-8fb0-55b85e275a36","Type":"ContainerStarted","Data":"415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e"}
Nov 03 18:40:23.212418 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.212273    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d" event={"ID":"866ab07a-6408-4046-8fb0-55b85e275a36","Type":"ContainerStarted","Data":"4a457d2896f1d0b3626c17a728956ffdeb3bc1317f30972bef574652eb57e430"}
Nov 03 18:40:23.212418 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.212309    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:23.213992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.213964    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf" event={"ID":"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2","Type":"ContainerStarted","Data":"7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc"}
Nov 03 18:40:23.213992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.214005    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf" event={"ID":"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2","Type":"ContainerStarted","Data":"aae0a92eb917a15e88798aa35fc10fcf086ffed48d8d424bad97212976bb3d9f"}
Nov 03 18:40:23.214205 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.214045    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:23.330983 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.330914    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d" podStartSLOduration=1.330896734 podStartE2EDuration="1.330896734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:23.330542152 +0000 UTC m=+3959.341280226" watchObservedRunningTime="2026-11-03 18:40:23.330896734 +0000 UTC m=+3959.341634798"
Nov 03 18:40:23.450706 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.450642    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf" podStartSLOduration=1.450622519 podStartE2EDuration="1.450622519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:23.448902218 +0000 UTC m=+3959.459640287" watchObservedRunningTime="2026-11-03 18:40:23.450622519 +0000 UTC m=+3959.461360591"
Nov 03 18:40:23.565378 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.565343    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"]
Nov 03 18:40:23.570398 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.570366    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.625398 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.625361    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.625614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.625412    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnk8k\" (UniqueName: \"kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.625614 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.625594    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.625791 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.625762    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.628751 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.628699    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c188b2e5-serving-cert\""
Nov 03 18:40:23.630311 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.630288    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"]
Nov 03 18:40:23.727220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.727121    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.727220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.727211    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.727429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.727257    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnk8k\" (UniqueName: \"kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.727429 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:40:23.727298    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-serving-cert: secret "maas-api-e2e-ait-c188b2e5-serving-cert" not found
Nov 03 18:40:23.727429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.727323    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.727429 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:40:23.727389    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls podName:805867c8-7676-41ab-853b-b28713aed8c6 nodeName:}" failed. No retries permitted until 2026-11-03 18:40:24.227365635 +0000 UTC m=+3960.238103701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls") pod "maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" (UID: "805867c8-7676-41ab-853b-b28713aed8c6") : secret "maas-api-e2e-ait-c188b2e5-serving-cert" not found
Nov 03 18:40:23.727959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.727934    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.730347 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.730319    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.777619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.777570    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:23.792013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.791986    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnk8k\" (UniqueName: \"kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:23.935118 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:23.935085    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:24.161232 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:24.161195    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:24.232027 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:24.231992    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:24.234592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:24.234571    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\") pod \"maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:24.487753 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:24.487647    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:24.871246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:24.871223    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"]
Nov 03 18:40:24.873452 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:40:24.873412    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805867c8_7676_41ab_853b_b28713aed8c6.slice/crio-72803faa1b7a72401cbac0f24e53f748ebf56c6a920aa6315433b82e647c4b42 WatchSource:0}: Error finding container 72803faa1b7a72401cbac0f24e53f748ebf56c6a920aa6315433b82e647c4b42: Status 404 returned error can't find the container with id 72803faa1b7a72401cbac0f24e53f748ebf56c6a920aa6315433b82e647c4b42
Nov 03 18:40:25.227295 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:25.227258    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" event={"ID":"805867c8-7676-41ab-853b-b28713aed8c6","Type":"ContainerStarted","Data":"72803faa1b7a72401cbac0f24e53f748ebf56c6a920aa6315433b82e647c4b42"}
Nov 03 18:40:26.235551 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:26.235514    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" event={"ID":"805867c8-7676-41ab-853b-b28713aed8c6","Type":"ContainerStarted","Data":"8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3"}
Nov 03 18:40:26.236011 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:26.235613    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:26.358120 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:26.358066    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" podStartSLOduration=2.548015694 podStartE2EDuration="3.358051317s" totalImagesPullingTime="810.035623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:26.356351989 +0000 UTC m=+3962.367090062" watchObservedRunningTime="2026-11-03 18:40:26.358051317 +0000 UTC m=+3962.368789388"
Nov 03 18:40:28.234101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:28.234066    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:29.207476 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:29.207432    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:32.170491 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:32.170455    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:40:32.249241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:32.249209    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:40:34.223350 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:34.223321    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:40:34.223769 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:34.223394    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:40:34.330667 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:34.330631    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:39.406404 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:39.406366    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:40.341002 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:40:40.340961    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:40:44.861352 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:44.861316    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:50.334230 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:50.334195    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:53.682284 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:53.682245    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:55.384102 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:40:55.384066    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:00.389620 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:00.389585    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:05.489928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:05.489891    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:06.707680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:06.707646    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"]
Nov 03 18:41:06.816324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:06.816275    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"]
Nov 03 18:41:07.079445 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:07.079366    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:07.237710 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:07.237671    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:10.339181 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:10.339142    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:41:10.497680 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:10.497644    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:11.709018 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:11.708945    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d" podUID="866ab07a-6408-4046-8fb0-55b85e275a36" containerName="payload-processing" containerID="cri-o://415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e" gracePeriod=25
Nov 03 18:41:11.817499 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:11.817437    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf" podUID="e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" containerName="payload-pre-processing" containerID="cri-o://7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc" gracePeriod=25
Nov 03 18:41:11.975519 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:11.975446    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:41:12.080157 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.080126    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:41:12.149510 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.149466    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v\" (UniqueName: \"kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v\") pod \"866ab07a-6408-4046-8fb0-55b85e275a36\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") "
Nov 03 18:41:12.149694 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.149653    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume\") pod \"866ab07a-6408-4046-8fb0-55b85e275a36\" (UID: \"866ab07a-6408-4046-8fb0-55b85e275a36\") "
Nov 03 18:41:12.150072 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.150041    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume" pod "866ab07a-6408-4046-8fb0-55b85e275a36" (UID: "866ab07a-6408-4046-8fb0-55b85e275a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:41:12.151918 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.151897    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v" pod "866ab07a-6408-4046-8fb0-55b85e275a36" (UID: "866ab07a-6408-4046-8fb0-55b85e275a36"). InnerVolumeSpecName "kube-api-access-mjp8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:41:12.250495 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.250410    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume\") pod \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") "
Nov 03 18:41:12.250495 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.250479    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2\" (UniqueName: \"kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2\") pod \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\" (UID: \"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2\") "
Nov 03 18:41:12.250839 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.250809    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume" pod "e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" (UID: "e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:41:12.250973 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.250951    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866ab07a-6408-4046-8fb0-55b85e275a36-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:12.251028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.250971    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:12.251028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.250986    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjp8v\" (UniqueName: \"kubernetes.io/projected/866ab07a-6408-4046-8fb0-55b85e275a36-kube-api-access-mjp8v\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:12.252924 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.252900    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2" pod "e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" (UID: "e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2"). InnerVolumeSpecName "kube-api-access-hlbl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:41:12.351907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.351861    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbl2\" (UniqueName: \"kubernetes.io/projected/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2-kube-api-access-hlbl2\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:12.524281 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.524187    5838 generic.go:356] "Generic (PLEG): container finished" podID="e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" containerID="7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc" exitCode=0
Nov 03 18:41:12.524281 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.524233    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"
Nov 03 18:41:12.524541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.524275    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf" event={"ID":"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2","Type":"ContainerDied","Data":"7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc"}
Nov 03 18:41:12.524541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.524333    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf" event={"ID":"e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2","Type":"ContainerDied","Data":"aae0a92eb917a15e88798aa35fc10fcf086ffed48d8d424bad97212976bb3d9f"}
Nov 03 18:41:12.524541 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.524354    5838 scope.go:122] "RemoveContainer" containerID="7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc"
Nov 03 18:41:12.525997 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.525968    5838 generic.go:356] "Generic (PLEG): container finished" podID="866ab07a-6408-4046-8fb0-55b85e275a36" containerID="415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e" exitCode=0
Nov 03 18:41:12.526128 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.526053    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d" event={"ID":"866ab07a-6408-4046-8fb0-55b85e275a36","Type":"ContainerDied","Data":"415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e"}
Nov 03 18:41:12.526128 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.526093    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d" event={"ID":"866ab07a-6408-4046-8fb0-55b85e275a36","Type":"ContainerDied","Data":"4a457d2896f1d0b3626c17a728956ffdeb3bc1317f30972bef574652eb57e430"}
Nov 03 18:41:12.526220 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.526183    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"
Nov 03 18:41:12.536347 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.536327    5838 scope.go:122] "RemoveContainer" containerID="7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc"
Nov 03 18:41:12.536641 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:12.536619    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc\": container with ID starting with 7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc not found: ID does not exist" containerID="7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc"
Nov 03 18:41:12.536723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.536655    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc"} err="failed to get container status \"7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc\": rpc error: code = NotFound desc = could not find container \"7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc\": container with ID starting with 7c8485bb5ebf4651da786e50e0c355038dd9fe4f8e292f4d4edbe097af2917cc not found: ID does not exist"
Nov 03 18:41:12.536723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.536682    5838 scope.go:122] "RemoveContainer" containerID="415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e"
Nov 03 18:41:12.545695 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.545673    5838 scope.go:122] "RemoveContainer" containerID="415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e"
Nov 03 18:41:12.546006 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:12.545981    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e\": container with ID starting with 415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e not found: ID does not exist" containerID="415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e"
Nov 03 18:41:12.546068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.546013    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e"} err="failed to get container status \"415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e\": rpc error: code = NotFound desc = could not find container \"415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e\": container with ID starting with 415293e83c03f3fe93ab1a7a6380e29e8715307d283f2270528f4672ad7d8f5e not found: ID does not exist"
Nov 03 18:41:12.693070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.693032    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"]
Nov 03 18:41:12.698843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.698816    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c188b2e5-55b886dc6c-vsm7d"]
Nov 03 18:41:12.811125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.811036    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"]
Nov 03 18:41:12.818633 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:12.818607    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c188b2e5-9ccb5bcf9-9d8xf"]
Nov 03 18:41:14.116044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:14.116009    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866ab07a-6408-4046-8fb0-55b85e275a36" path="/var/lib/kubelet/pods/866ab07a-6408-4046-8fb0-55b85e275a36/volumes"
Nov 03 18:41:14.116643 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:14.116621    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" path="/var/lib/kubelet/pods/e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2/volumes"
Nov 03 18:41:14.835708 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:14.835665    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"]
Nov 03 18:41:14.835984 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:14.835952    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" podUID="805867c8-7676-41ab-853b-b28713aed8c6" containerName="maas-api" containerID="cri-o://8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3" gracePeriod=30
Nov 03 18:41:14.938938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:14.938900    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:15.108010 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.107981    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:41:15.281768 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.281702    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs\") pod \"805867c8-7676-41ab-853b-b28713aed8c6\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") "
Nov 03 18:41:15.282208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.281808    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k\" (UniqueName: \"kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k\") pod \"805867c8-7676-41ab-853b-b28713aed8c6\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") "
Nov 03 18:41:15.282208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.281856    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\") pod \"805867c8-7676-41ab-853b-b28713aed8c6\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") "
Nov 03 18:41:15.282208 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.281894    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca\") pod \"805867c8-7676-41ab-853b-b28713aed8c6\" (UID: \"805867c8-7676-41ab-853b-b28713aed8c6\") "
Nov 03 18:41:15.282394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.282319    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca" pod "805867c8-7676-41ab-853b-b28713aed8c6" (UID: "805867c8-7676-41ab-853b-b28713aed8c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:41:15.284084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.284049    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs" pod "805867c8-7676-41ab-853b-b28713aed8c6" (UID: "805867c8-7676-41ab-853b-b28713aed8c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:41:15.284084 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.284074    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls" pod "805867c8-7676-41ab-853b-b28713aed8c6" (UID: "805867c8-7676-41ab-853b-b28713aed8c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:41:15.284301 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.284109    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k" pod "805867c8-7676-41ab-853b-b28713aed8c6" (UID: "805867c8-7676-41ab-853b-b28713aed8c6"). InnerVolumeSpecName "kube-api-access-hnk8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:41:15.383666 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.383562    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnk8k\" (UniqueName: \"kubernetes.io/projected/805867c8-7676-41ab-853b-b28713aed8c6-kube-api-access-hnk8k\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:15.383666 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.383601    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:15.383666 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.383615    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805867c8-7676-41ab-853b-b28713aed8c6-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:15.383666 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.383627    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805867c8-7676-41ab-853b-b28713aed8c6-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:41:15.548375 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.548337    5838 generic.go:356] "Generic (PLEG): container finished" podID="805867c8-7676-41ab-853b-b28713aed8c6" containerID="8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3" exitCode=0
Nov 03 18:41:15.548608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.548402    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"
Nov 03 18:41:15.548608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.548426    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" event={"ID":"805867c8-7676-41ab-853b-b28713aed8c6","Type":"ContainerDied","Data":"8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3"}
Nov 03 18:41:15.548608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.548533    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz" event={"ID":"805867c8-7676-41ab-853b-b28713aed8c6","Type":"ContainerDied","Data":"72803faa1b7a72401cbac0f24e53f748ebf56c6a920aa6315433b82e647c4b42"}
Nov 03 18:41:15.548608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.548573    5838 scope.go:122] "RemoveContainer" containerID="8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3"
Nov 03 18:41:15.559785 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.559765    5838 scope.go:122] "RemoveContainer" containerID="8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3"
Nov 03 18:41:15.560087 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:15.560061    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3\": container with ID starting with 8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3 not found: ID does not exist" containerID="8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3"
Nov 03 18:41:15.560169 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.560099    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3"} err="failed to get container status \"8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3\": rpc error: code = NotFound desc = could not find container \"8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3\": container with ID starting with 8eeb99e88906959dd5c52cd017f72e63cf550d532b81d00420d4e335625973a3 not found: ID does not exist"
Nov 03 18:41:15.717992 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.717957    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"]
Nov 03 18:41:15.723750 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:15.723690    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c188b2e5-5b97d67859-ws8rz"]
Nov 03 18:41:16.116939 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:16.116860    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805867c8-7676-41ab-853b-b28713aed8c6" path="/var/lib/kubelet/pods/805867c8-7676-41ab-853b-b28713aed8c6/volumes"
Nov 03 18:41:16.550977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:16.550940    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:16.842044 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:16.841943    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:19.462603 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:19.462569    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:41:21.477953 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:21.477920    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:22.661758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:22.661691    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:23.316194 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:23.316154    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:23.699264 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:23.699223    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:27.503286 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:27.503235    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:27.676199 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:27.676166    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:27.833990 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:27.833912    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:30.780830 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:30.780790    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:30.958031 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:30.957998    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:31.330721 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.330668    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"]
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331616    5838 cpu_manager.go:414] "containerMap: removing container" podUID="866ab07a-6408-4046-8fb0-55b85e275a36" containerName="payload-processing"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331639    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866ab07a-6408-4046-8fb0-55b85e275a36" containerName="payload-processing"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331720    5838 cpu_manager.go:414] "containerMap: removing container" podUID="e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" containerName="payload-pre-processing"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331754    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" containerName="payload-pre-processing"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331779    5838 cpu_manager.go:414] "containerMap: removing container" podUID="805867c8-7676-41ab-853b-b28713aed8c6" containerName="maas-api"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331788    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805867c8-7676-41ab-853b-b28713aed8c6" containerName="maas-api"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331925    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="805867c8-7676-41ab-853b-b28713aed8c6" containerName="maas-api"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331942    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="866ab07a-6408-4046-8fb0-55b85e275a36" containerName="payload-processing"
Nov 03 18:41:31.333137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.331958    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d269d6-6bdb-4c38-b1eb-d0fd0ee6e3a2" containerName="payload-pre-processing"
Nov 03 18:41:31.344092 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.344056    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.398097 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.398048    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"]
Nov 03 18:41:31.445148 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.445116    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-8d4263\""
Nov 03 18:41:31.445345 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.445241    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-8d4263-dockercfg-94zsz\""
Nov 03 18:41:31.449678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.449653    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.449850 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.449751    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckxn\" (UniqueName: \"kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn\") pod \"payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.550786 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.550750    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.551011 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.550811    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckxn\" (UniqueName: \"kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn\") pod \"payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.551450 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.551424    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.614246 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.614171    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckxn\" (UniqueName: \"kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn\") pod \"payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:31.666584 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:31.666550    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:32.046843 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:32.046814    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"]
Nov 03 18:41:32.048677 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:41:32.048639    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1de3ce4_d9e0_409f_818f_9055fb323594.slice/crio-2169064157ec7f37c7a51e806f8f5d4816712f3e33df0c7672f4a0ac0b303e5f WatchSource:0}: Error finding container 2169064157ec7f37c7a51e806f8f5d4816712f3e33df0c7672f4a0ac0b303e5f: Status 404 returned error can't find the container with id 2169064157ec7f37c7a51e806f8f5d4816712f3e33df0c7672f4a0ac0b303e5f
Nov 03 18:41:32.665875 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:32.665825    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j" event={"ID":"d1de3ce4-d9e0-409f-818f-9055fb323594","Type":"ContainerStarted","Data":"756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20"}
Nov 03 18:41:32.666066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:32.665919    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j" event={"ID":"d1de3ce4-d9e0-409f-818f-9055fb323594","Type":"ContainerStarted","Data":"2169064157ec7f37c7a51e806f8f5d4816712f3e33df0c7672f4a0ac0b303e5f"}
Nov 03 18:41:32.666066 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:32.665947    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:32.785718 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:32.785662    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j" podStartSLOduration=1.7856459340000002 podStartE2EDuration="1.785645934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:41:32.783963976 +0000 UTC m=+4028.794702050" watchObservedRunningTime="2026-11-03 18:41:32.785645934 +0000 UTC m=+4028.796384005"
Nov 03 18:41:33.634292 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.634255    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"]
Nov 03 18:41:33.638137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.638119    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.671956 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.671921    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mf7\" (UniqueName: \"kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.672184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.671971    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.672184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.672137    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.672184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.672173    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.692807 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.692782    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-8d4263-serving-cert\""
Nov 03 18:41:33.699724 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.699696    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"]
Nov 03 18:41:33.773545 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.773512    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.773726 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.773663    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mf7\" (UniqueName: \"kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.773726 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.773698    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.773891 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:33.773798    5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-serving-cert: secret "maas-api-e2e-authcfg-http-8d4263-serving-cert" not found
Nov 03 18:41:33.773891 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:33.773853    5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls podName:64b667f1-2530-41bd-a27c-5cd321e69f56 nodeName:}" failed. No retries permitted until 2026-11-03 18:41:34.27383707 +0000 UTC m=+4030.284575121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls") pod "maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" (UID: "64b667f1-2530-41bd-a27c-5cd321e69f56") : secret "maas-api-e2e-authcfg-http-8d4263-serving-cert" not found
Nov 03 18:41:33.774006 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.773964    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.774580 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.774554    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.776594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.776565    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.830046 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.830010    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:33.836618 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.836593    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mf7\" (UniqueName: \"kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:33.984392 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:33.984351    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:34.141282 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:34.141247    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:34.279723 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:34.279625    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:34.282303 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:34.282273    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:34.552457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:34.552367    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:34.937964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:34.937934    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"]
Nov 03 18:41:34.939815 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:41:34.939789    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b667f1_2530_41bd_a27c_5cd321e69f56.slice/crio-771c062f4ec0847c8e48248c2bd81bc38591775511e372d714dd9fb1cef5d67a WatchSource:0}: Error finding container 771c062f4ec0847c8e48248c2bd81bc38591775511e372d714dd9fb1cef5d67a: Status 404 returned error can't find the container with id 771c062f4ec0847c8e48248c2bd81bc38591775511e372d714dd9fb1cef5d67a
Nov 03 18:41:35.686581 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:35.686533    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" event={"ID":"64b667f1-2530-41bd-a27c-5cd321e69f56","Type":"ContainerStarted","Data":"771c062f4ec0847c8e48248c2bd81bc38591775511e372d714dd9fb1cef5d67a"}
Nov 03 18:41:36.694236 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:36.694199    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" event={"ID":"64b667f1-2530-41bd-a27c-5cd321e69f56","Type":"ContainerStarted","Data":"52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355"}
Nov 03 18:41:36.694666 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:36.694286    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:36.815676 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:36.815622    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" podStartSLOduration=2.8757298540000003 podStartE2EDuration="3.81560715s" totalImagesPullingTime="939.877296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:41:36.81480829 +0000 UTC m=+4032.825546383" watchObservedRunningTime="2026-11-03 18:41:36.81560715 +0000 UTC m=+4032.826345219"
Nov 03 18:41:39.031603 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:39.031563    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:40.340039 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:41:40.339997    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:41:42.708512 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:42.708471    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:41:43.127917 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:43.127836    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:41:43.676543 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:43.676514    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:41:44.025966 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:44.025874    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:49.128132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:49.128093    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:52.367229 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:52.367195    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:54.028115 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:54.028071    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:54.522611 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:54.522582    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:59.603058 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:41:59.603021    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:04.681800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:04.681702    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:09.698872 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:09.698829    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:10.338845 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:42:10.338804    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:42:14.350101 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:14.350055    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:14.706301 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:14.706267    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:19.690874 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:19.690834    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:19.848631 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:19.848591    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:24.794144 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:24.794106    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:25.729824 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:25.729789    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:28.749920 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:28.749886    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:28.901075 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:28.901038    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:29.791391 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:29.791353    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:34.810619 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:34.810577    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:39.792626 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:39.792590    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:40.339066 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:42:40.339024    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:42:44.902765 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:44.902663    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:46.576861 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:46.576828    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:49.995196 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:49.995156    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:54.378137 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:54.378096    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:55.082028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:42:55.081992    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:00.047948 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:00.047909    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:00.206013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:00.205979    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:05.195408 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:05.195374    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:10.194071 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:10.194035    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:10.338961 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:43:10.338921    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:43:15.536941 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:15.536904    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:20.683515 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:20.683477    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:24.404094 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:24.404057    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"]
Nov 03 18:43:24.774488 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:24.774397    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:24.929577 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:24.929545    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:29.405024 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.404984    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j" podUID="d1de3ce4-d9e0-409f-818f-9055fb323594" containerName="payload-processing" containerID="cri-o://756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20" gracePeriod=25
Nov 03 18:43:29.668678 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.668618    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:43:29.791184 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.791153    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume\") pod \"d1de3ce4-d9e0-409f-818f-9055fb323594\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") "
Nov 03 18:43:29.791374 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.791273    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn\" (UniqueName: \"kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn\") pod \"d1de3ce4-d9e0-409f-818f-9055fb323594\" (UID: \"d1de3ce4-d9e0-409f-818f-9055fb323594\") "
Nov 03 18:43:29.791539 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.791515    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume" pod "d1de3ce4-d9e0-409f-818f-9055fb323594" (UID: "d1de3ce4-d9e0-409f-818f-9055fb323594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:43:29.791696 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.791673    5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1de3ce4-d9e0-409f-818f-9055fb323594-plugins-config-volume\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:29.794153 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.794123    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn" pod "d1de3ce4-d9e0-409f-818f-9055fb323594" (UID: "d1de3ce4-d9e0-409f-818f-9055fb323594"). InnerVolumeSpecName "kube-api-access-cckxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:43:29.892828 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:29.892794    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckxn\" (UniqueName: \"kubernetes.io/projected/d1de3ce4-d9e0-409f-818f-9055fb323594-kube-api-access-cckxn\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:30.409250 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.409209    5838 generic.go:356] "Generic (PLEG): container finished" podID="d1de3ce4-d9e0-409f-818f-9055fb323594" containerID="756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20" exitCode=0
Nov 03 18:43:30.409250 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.409252    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"
Nov 03 18:43:30.409860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.409303    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j" event={"ID":"d1de3ce4-d9e0-409f-818f-9055fb323594","Type":"ContainerDied","Data":"756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20"}
Nov 03 18:43:30.409860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.409351    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j" event={"ID":"d1de3ce4-d9e0-409f-818f-9055fb323594","Type":"ContainerDied","Data":"2169064157ec7f37c7a51e806f8f5d4816712f3e33df0c7672f4a0ac0b303e5f"}
Nov 03 18:43:30.409860 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.409375    5838 scope.go:122] "RemoveContainer" containerID="756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20"
Nov 03 18:43:30.419800 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.419776    5838 scope.go:122] "RemoveContainer" containerID="756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20"
Nov 03 18:43:30.420126 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:43:30.420102    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20\": container with ID starting with 756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20 not found: ID does not exist" containerID="756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20"
Nov 03 18:43:30.420189 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.420135    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20"} err="failed to get container status \"756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20\": rpc error: code = NotFound desc = could not find container \"756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20\": container with ID starting with 756bbc6627ed8f79e4e7937fc5c08aac4fc9e5fc68175f75b75f84757359ac20 not found: ID does not exist"
Nov 03 18:43:30.578438 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.578405    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"]
Nov 03 18:43:30.584132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:30.584105    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d4263-f776bc5d5-n4x8j"]
Nov 03 18:43:32.115652 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:32.115615    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1de3ce4-d9e0-409f-818f-9055fb323594" path="/var/lib/kubelet/pods/d1de3ce4-d9e0-409f-818f-9055fb323594/volumes"
Nov 03 18:43:34.782052 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:34.781961    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"]
Nov 03 18:43:34.782544 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:34.782227    5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" podUID="64b667f1-2530-41bd-a27c-5cd321e69f56" containerName="maas-api" containerID="cri-o://52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355" gracePeriod=30
Nov 03 18:43:34.888092 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:34.888059    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:35.050487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.050459    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:43:35.149440 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.149394    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca\") pod \"64b667f1-2530-41bd-a27c-5cd321e69f56\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") "
Nov 03 18:43:35.149649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.149519    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs\") pod \"64b667f1-2530-41bd-a27c-5cd321e69f56\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") "
Nov 03 18:43:35.149649 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.149624    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7\" (UniqueName: \"kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7\") pod \"64b667f1-2530-41bd-a27c-5cd321e69f56\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") "
Nov 03 18:43:35.149815 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.149661    5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\") pod \"64b667f1-2530-41bd-a27c-5cd321e69f56\" (UID: \"64b667f1-2530-41bd-a27c-5cd321e69f56\") "
Nov 03 18:43:35.149872 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.149827    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca" pod "64b667f1-2530-41bd-a27c-5cd321e69f56" (UID: "64b667f1-2530-41bd-a27c-5cd321e69f56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:43:35.150082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.150063    5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b667f1-2530-41bd-a27c-5cd321e69f56-openshift-service-ca\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:35.151907 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.151875    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls" pod "64b667f1-2530-41bd-a27c-5cd321e69f56" (UID: "64b667f1-2530-41bd-a27c-5cd321e69f56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:43:35.152031 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.151933    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs" pod "64b667f1-2530-41bd-a27c-5cd321e69f56" (UID: "64b667f1-2530-41bd-a27c-5cd321e69f56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:43:35.152031 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.152000    5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7" pod "64b667f1-2530-41bd-a27c-5cd321e69f56" (UID: "64b667f1-2530-41bd-a27c-5cd321e69f56"). InnerVolumeSpecName "kube-api-access-v6mf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:43:35.250699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.250655    5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-metrics-certs\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:35.250699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.250687    5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mf7\" (UniqueName: \"kubernetes.io/projected/64b667f1-2530-41bd-a27c-5cd321e69f56-kube-api-access-v6mf7\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:35.250699 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.250701    5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b667f1-2530-41bd-a27c-5cd321e69f56-maas-api-tls\") on node \"ip-10-0-139-156.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:35.442486 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.442449    5838 generic.go:356] "Generic (PLEG): container finished" podID="64b667f1-2530-41bd-a27c-5cd321e69f56" containerID="52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355" exitCode=0
Nov 03 18:43:35.442692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.442494    5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"
Nov 03 18:43:35.442692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.442544    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" event={"ID":"64b667f1-2530-41bd-a27c-5cd321e69f56","Type":"ContainerDied","Data":"52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355"}
Nov 03 18:43:35.442692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.442604    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6" event={"ID":"64b667f1-2530-41bd-a27c-5cd321e69f56","Type":"ContainerDied","Data":"771c062f4ec0847c8e48248c2bd81bc38591775511e372d714dd9fb1cef5d67a"}
Nov 03 18:43:35.442692 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.442630    5838 scope.go:122] "RemoveContainer" containerID="52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355"
Nov 03 18:43:35.456241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.456216    5838 scope.go:122] "RemoveContainer" containerID="52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355"
Nov 03 18:43:35.456567 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:43:35.456545    5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355\": container with ID starting with 52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355 not found: ID does not exist" containerID="52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355"
Nov 03 18:43:35.456645 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.456576    5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355"} err="failed to get container status \"52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355\": rpc error: code = NotFound desc = could not find container \"52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355\": container with ID starting with 52d3d68c74ecee13ce390a9adf5e6be5e24bf3b4de0296733056c4693fbbd355 not found: ID does not exist"
Nov 03 18:43:35.612243 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.612209    5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"]
Nov 03 18:43:35.618104 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:35.618072    5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d4263-759869c549-fjgr6"]
Nov 03 18:43:36.115534 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:36.115503    5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b667f1-2530-41bd-a27c-5cd321e69f56" path="/var/lib/kubelet/pods/64b667f1-2530-41bd-a27c-5cd321e69f56/volumes"
Nov 03 18:43:36.431273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:36.431223    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:36.740029 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:36.739942    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:37.500020 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:37.499981    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:39.446963 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:39.446926    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:43:40.340437 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:43:40.340394    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:43:44.939143 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:44.939008    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:45.096511 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:45.096461    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:45.309404 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:45.309320    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:53.235242 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:53.235198    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:43:54.762120 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:43:54.762086    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:04.724180 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:04.724144    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:44:07.375762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:07.375105    5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8jw2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:44:07.375762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:07.375224    5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8jw2g" podUID="aaff885c-bbb0-43e2-92f7-97a48d101d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:44:10.338221 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:44:10.338181    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:44:25.123252 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:25.123215    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:44:25.135806 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:25.135776    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hwxhd_5908cbf5-2e94-4a1c-9c81-f7adfd472da0/ovn-acl-logging/0.log"
Nov 03 18:44:25.927582 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:25.927539    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:44:40.338306 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:44:40.338260    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:44:55.169596 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:55.169556    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:56.083583 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:56.083544    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:59.842592 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:44:59.842548    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-84n8v"]
Nov 03 18:45:10.229043 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:10.228943    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-788f8bd568-8vqvp_edeb0b48-390f-43f0-9e5b-e14a7ff76626/manager/0.log"
Nov 03 18:45:10.338712 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:45:10.338668    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:45:10.594621 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:10.594536    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g6hps_48de8580-21aa-40d3-add8-03bcdfdeba4d/manager/0.log"
Nov 03 18:45:13.796336 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:13.796296    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7456bf56b5-kmlfq_c2b4d845-94ca-4ce6-847c-eeccad6ce534/maas-api/0.log"
Nov 03 18:45:17.208028 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:17.207991    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z8zbr_ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0/manager/0.log"
Nov 03 18:45:17.537670 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:17.537564    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9l9z8_146565e0-81b4-4c59-8505-b8f99bcfc8eb/manager/0.log"
Nov 03 18:45:17.761309 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:17.761280    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-76xkn_0bdb49a9-4db8-4775-adbf-cd1e1779dbb2/kuadrant-console-plugin/0.log"
Nov 03 18:45:18.046390 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:18.046354    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-c8sr8_67d70366-124c-4a0a-ac40-76957d0331e5/manager/0.log"
Nov 03 18:45:18.474599 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:18.474569    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-84n8v_0456b517-d0a7-4fee-b0f3-635cbad921e7/limitador/0.log"
Nov 03 18:45:18.716435 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:18.716400    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rblzq_29ba8591-1315-4eea-bf6a-1c81882666e7/manager/0.log"
Nov 03 18:45:19.543959 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:19.543930    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f85rq9ds_c815c4f2-540a-4951-910a-b75c9ad9a72d/istio-proxy/0.log"
Nov 03 18:45:21.210976 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:21.210948    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-8jw2g_aaff885c-bbb0-43e2-92f7-97a48d101d52/istio-proxy/0.log"
Nov 03 18:45:21.697859 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:21.697827    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-jncwc_1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0/istio-proxy/0.log"
Nov 03 18:45:22.219787 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:22.219754    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-ml2hv_efc862fa-db10-4720-9d9b-90d4533a216c/istio-proxy/0.log"
Nov 03 18:45:23.360090 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:23.360059    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx_7cad14b8-057f-416d-938c-505f90358b9a/istio-proxy/0.log"
Nov 03 18:45:23.585108 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:23.585075    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gmtqs_1b848dc6-7e5e-46c2-b1e6-63fa37e718ac/payload-pre-processing/0.log"
Nov 03 18:45:23.800106 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:23.800077    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-46szt_25ceda13-c002-4ee6-9926-68b0e3a98cb9/payload-processing/0.log"
Nov 03 18:45:25.003414 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:25.003379    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj_00ed915e-64f7-4efd-981b-71d7c8756b48/main/0.log"
Nov 03 18:45:25.275974 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:25.275888    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qrswt_f956514d-9307-413e-a516-7d5449040eb8/main/0.log"
Nov 03 18:45:25.514766 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:25.514715    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-plqxs_6ebaa47f-2620-4fb8-8843-eb8419cf4c2a/main/0.log"
Nov 03 18:45:37.183394 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:37.183357    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-788f8bd568-8vqvp_edeb0b48-390f-43f0-9e5b-e14a7ff76626/manager/0.log"
Nov 03 18:45:37.553241 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:37.553149    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g6hps_48de8580-21aa-40d3-add8-03bcdfdeba4d/manager/0.log"
Nov 03 18:45:40.307629 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:40.307593    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7456bf56b5-kmlfq_c2b4d845-94ca-4ce6-847c-eeccad6ce534/maas-api/0.log"
Nov 03 18:45:40.338801 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:45:40.338759    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:45:43.036273 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:43.036238    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z8zbr_ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0/manager/0.log"
Nov 03 18:45:43.365211 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:43.365137    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9l9z8_146565e0-81b4-4c59-8505-b8f99bcfc8eb/manager/0.log"
Nov 03 18:45:43.588594 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:43.588563    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-76xkn_0bdb49a9-4db8-4775-adbf-cd1e1779dbb2/kuadrant-console-plugin/0.log"
Nov 03 18:45:43.831058 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:43.831004    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-c8sr8_67d70366-124c-4a0a-ac40-76957d0331e5/manager/0.log"
Nov 03 18:45:44.118303 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:44.118202    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-84n8v_0456b517-d0a7-4fee-b0f3-635cbad921e7/limitador/0.log"
Nov 03 18:45:44.352939 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:44.352893    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rblzq_29ba8591-1315-4eea-bf6a-1c81882666e7/manager/0.log"
Nov 03 18:45:44.973526 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:44.973424    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f85rq9ds_c815c4f2-540a-4951-910a-b75c9ad9a72d/istio-proxy/0.log"
Nov 03 18:45:46.595373 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:46.595340    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-8jw2g_aaff885c-bbb0-43e2-92f7-97a48d101d52/istio-proxy/0.log"
Nov 03 18:45:47.095049 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:47.095018    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-jncwc_1647bd7a-6878-40ad-b6b2-c4e06dcd9cc0/istio-proxy/0.log"
Nov 03 18:45:47.609112 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:47.609085    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-ml2hv_efc862fa-db10-4720-9d9b-90d4533a216c/istio-proxy/0.log"
Nov 03 18:45:48.684529 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:48.684500    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-28fmx_7cad14b8-057f-416d-938c-505f90358b9a/istio-proxy/0.log"
Nov 03 18:45:48.907977 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:48.907931    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gmtqs_1b848dc6-7e5e-46c2-b1e6-63fa37e718ac/payload-pre-processing/0.log"
Nov 03 18:45:49.130898 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:49.130816    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-46szt_25ceda13-c002-4ee6-9926-68b0e3a98cb9/payload-processing/0.log"
Nov 03 18:45:50.052218 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:50.052188    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xmrnj_00ed915e-64f7-4efd-981b-71d7c8756b48/main/0.log"
Nov 03 18:45:50.283110 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:50.283082    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qrswt_f956514d-9307-413e-a516-7d5449040eb8/main/0.log"
Nov 03 18:45:50.514176 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:50.514144    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-plqxs_6ebaa47f-2620-4fb8-8843-eb8419cf4c2a/main/0.log"
Nov 03 18:45:55.465068 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:45:55.465031    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:46:10.339195 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:46:10.339153    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:46:12.214746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:46:12.214703    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:46:12.367048 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:46:12.367006    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:46:27.216013 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:46:27.215979    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-46szt_25ceda13-c002-4ee6-9926-68b0e3a98cb9/payload-processing/0.log"
Nov 03 18:46:27.449925 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:46:27.449896    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-46szt_25ceda13-c002-4ee6-9926-68b0e3a98cb9/payload-processing/0.log"
Nov 03 18:46:40.338789 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:46:40.338745    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:46:55.810896 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:46:55.810859    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:06.449698 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:06.449664    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-46szt_25ceda13-c002-4ee6-9926-68b0e3a98cb9/payload-processing/0.log"
Nov 03 18:47:06.676154 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:06.676120    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-46szt_25ceda13-c002-4ee6-9926-68b0e3a98cb9/payload-processing/0.log"
Nov 03 18:47:07.040457 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.040422    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlnz7/must-gather-8845c"]
Nov 03 18:47:07.040928 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.040914    5838 cpu_manager.go:414] "containerMap: removing container" podUID="64b667f1-2530-41bd-a27c-5cd321e69f56" containerName="maas-api"
Nov 03 18:47:07.040986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.040929    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b667f1-2530-41bd-a27c-5cd321e69f56" containerName="maas-api"
Nov 03 18:47:07.040986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.040945    5838 cpu_manager.go:414] "containerMap: removing container" podUID="d1de3ce4-d9e0-409f-818f-9055fb323594" containerName="payload-processing"
Nov 03 18:47:07.040986 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.040950    5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1de3ce4-d9e0-409f-818f-9055fb323594" containerName="payload-processing"
Nov 03 18:47:07.041083 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.041019    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b667f1-2530-41bd-a27c-5cd321e69f56" containerName="maas-api"
Nov 03 18:47:07.041083 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.041028    5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1de3ce4-d9e0-409f-818f-9055fb323594" containerName="payload-processing"
Nov 03 18:47:07.044325 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.044307    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.061954 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.061924    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:07.098938 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.098908    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlnz7\"/\"kube-root-ca.crt\""
Nov 03 18:47:07.148191 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.148158    5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlnz7\"/\"openshift-service-ca.crt\""
Nov 03 18:47:07.148389 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.148232    5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vlnz7\"/\"default-dockercfg-ngffq\""
Nov 03 18:47:07.150570 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.150548    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/must-gather-8845c"]
Nov 03 18:47:07.171486 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.171456    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ldf\" (UniqueName: \"kubernetes.io/projected/ea016908-ebf0-44f7-a50e-cf18da98f029-kube-api-access-t5ldf\") pod \"must-gather-8845c\" (UID: \"ea016908-ebf0-44f7-a50e-cf18da98f029\") " pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.171643 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.171576    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea016908-ebf0-44f7-a50e-cf18da98f029-must-gather-output\") pod \"must-gather-8845c\" (UID: \"ea016908-ebf0-44f7-a50e-cf18da98f029\") " pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.222487 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.222459    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:07.273024 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.272976    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ldf\" (UniqueName: \"kubernetes.io/projected/ea016908-ebf0-44f7-a50e-cf18da98f029-kube-api-access-t5ldf\") pod \"must-gather-8845c\" (UID: \"ea016908-ebf0-44f7-a50e-cf18da98f029\") " pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.273224 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.273066    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea016908-ebf0-44f7-a50e-cf18da98f029-must-gather-output\") pod \"must-gather-8845c\" (UID: \"ea016908-ebf0-44f7-a50e-cf18da98f029\") " pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.273429 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.273410    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea016908-ebf0-44f7-a50e-cf18da98f029-must-gather-output\") pod \"must-gather-8845c\" (UID: \"ea016908-ebf0-44f7-a50e-cf18da98f029\") " pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.336501 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.336424    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ldf\" (UniqueName: \"kubernetes.io/projected/ea016908-ebf0-44f7-a50e-cf18da98f029-kube-api-access-t5ldf\") pod \"must-gather-8845c\" (UID: \"ea016908-ebf0-44f7-a50e-cf18da98f029\") " pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.359418 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.359384    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/must-gather-8845c"
Nov 03 18:47:07.737324 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.737297    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/must-gather-8845c"]
Nov 03 18:47:07.739117 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:47:07.739090    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea016908_ebf0_44f7_a50e_cf18da98f029.slice/crio-1726063ced7f9a62e0c96331d27d33b111412dc6265cb9e8ccee5c0a6ac6ab3a WatchSource:0}: Error finding container 1726063ced7f9a62e0c96331d27d33b111412dc6265cb9e8ccee5c0a6ac6ab3a: Status 404 returned error can't find the container with id 1726063ced7f9a62e0c96331d27d33b111412dc6265cb9e8ccee5c0a6ac6ab3a
Nov 03 18:47:07.741082 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.741064    5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:47:07.787997 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:07.787961    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/must-gather-8845c" event={"ID":"ea016908-ebf0-44f7-a50e-cf18da98f029","Type":"ContainerStarted","Data":"1726063ced7f9a62e0c96331d27d33b111412dc6265cb9e8ccee5c0a6ac6ab3a"}
Nov 03 18:47:09.803608 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:09.803532    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/must-gather-8845c" event={"ID":"ea016908-ebf0-44f7-a50e-cf18da98f029","Type":"ContainerStarted","Data":"72bda4ff5dfe8b404bdb69b819ffc584e1af080d10a95cb3dbbc7b5896727cd1"}
Nov 03 18:47:09.804125 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:09.803621    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/must-gather-8845c" event={"ID":"ea016908-ebf0-44f7-a50e-cf18da98f029","Type":"ContainerStarted","Data":"ded395a404f325992593db28a9d20e37c3425b7cd65cf9a65d83ed810708e5f7"}
Nov 03 18:47:09.934447 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:09.934379    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlnz7/must-gather-8845c" podStartSLOduration=2.776476104 podStartE2EDuration="3.934359656s" totalImagesPullingTime="1.157883552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:09.932975167 +0000 UTC m=+4365.943713241" watchObservedRunningTime="2026-11-03 18:47:09.934359656 +0000 UTC m=+4365.945097730"
Nov 03 18:47:10.338647 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:47:10.338608    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:47:16.738250 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:16.738210    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cz52j_46723e8e-0ca8-4209-9904-b35885a2f90f/global-pull-secret-syncer/0.log"
Nov 03 18:47:19.275758 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:19.275136    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log"
Nov 03 18:47:21.055357 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:21.055324    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-156.us-west-1.compute.internal_779eab1b0b29ccc85d3654a5ae82e6cd/haproxy/0.log"
Nov 03 18:47:26.392610 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.392574    5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"]
Nov 03 18:47:26.397231 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.397212    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.485517 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.485478    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"]
Nov 03 18:47:26.493426 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.493394    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2km4x\" (UniqueName: \"kubernetes.io/projected/1a89960a-953b-48f6-ac38-4e4cf699d821-kube-api-access-2km4x\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.493591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.493503    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-proc\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.493591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.493556    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-podres\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.493591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.493590    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-lib-modules\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.493795 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.493768    5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-sys\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.568296 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.568262    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z8zbr_ed03bf6a-2b81-4466-ba6b-6bc624d1f3c0/manager/0.log"
Nov 03 18:47:26.595431 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595397    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-proc\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595468    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-podres\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595517    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-lib-modules\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595549    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-proc\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595560    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-sys\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595617    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-sys\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595675    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-podres\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.595716 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595708    5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2km4x\" (UniqueName: \"kubernetes.io/projected/1a89960a-953b-48f6-ac38-4e4cf699d821-kube-api-access-2km4x\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.596167 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.595760    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a89960a-953b-48f6-ac38-4e4cf699d821-lib-modules\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.659444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.659329    5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2km4x\" (UniqueName: \"kubernetes.io/projected/1a89960a-953b-48f6-ac38-4e4cf699d821-kube-api-access-2km4x\") pod \"perf-node-gather-daemonset-nq5bw\" (UID: \"1a89960a-953b-48f6-ac38-4e4cf699d821\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.710769 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.710710    5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:26.995124 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:26.995034    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9l9z8_146565e0-81b4-4c59-8505-b8f99bcfc8eb/manager/0.log"
Nov 03 18:47:27.089438 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:27.089408    5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"]
Nov 03 18:47:27.091231 ip-10-0-139-156 kubenswrapper[5838]: W1103 18:47:27.091199    5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a89960a_953b_48f6_ac38_4e4cf699d821.slice/crio-f0736623109fd174ef83624b13da2b8061ee2717cdac8e0a1ebec53f869616ae WatchSource:0}: Error finding container f0736623109fd174ef83624b13da2b8061ee2717cdac8e0a1ebec53f869616ae: Status 404 returned error can't find the container with id f0736623109fd174ef83624b13da2b8061ee2717cdac8e0a1ebec53f869616ae
Nov 03 18:47:27.420527 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:27.420498    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-76xkn_0bdb49a9-4db8-4775-adbf-cd1e1779dbb2/kuadrant-console-plugin/0.log"
Nov 03 18:47:27.869297 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:27.869211    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-c8sr8_67d70366-124c-4a0a-ac40-76957d0331e5/manager/0.log"
Nov 03 18:47:28.018372 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:28.018335    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw" event={"ID":"1a89960a-953b-48f6-ac38-4e4cf699d821","Type":"ContainerStarted","Data":"3511d4514b4e9efa11c08d721583a17b47020b25ed3da5540320f0cbc0b4aa79"}
Nov 03 18:47:28.018601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:28.018398    5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw" event={"ID":"1a89960a-953b-48f6-ac38-4e4cf699d821","Type":"ContainerStarted","Data":"f0736623109fd174ef83624b13da2b8061ee2717cdac8e0a1ebec53f869616ae"}
Nov 03 18:47:28.018601 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:28.018539    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:28.372814 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:28.372782    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-84n8v_0456b517-d0a7-4fee-b0f3-635cbad921e7/limitador/0.log"
Nov 03 18:47:28.840591 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:28.840552    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rblzq_29ba8591-1315-4eea-bf6a-1c81882666e7/manager/0.log"
Nov 03 18:47:31.470427 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:31.470379    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/alertmanager/0.log"
Nov 03 18:47:31.891132 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:31.891101    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/config-reloader/0.log"
Nov 03 18:47:32.311746 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:32.311628    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/kube-rbac-proxy-web/0.log"
Nov 03 18:47:32.732885 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:32.732847    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/kube-rbac-proxy/0.log"
Nov 03 18:47:33.153701 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:33.153667    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/kube-rbac-proxy-metric/0.log"
Nov 03 18:47:33.578035 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:33.577954    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/prom-label-proxy/0.log"
Nov 03 18:47:33.999762 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:33.999718    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b802a022-e188-4411-806c-d8ccb0e22450/init-config-reloader/0.log"
Nov 03 18:47:34.041327 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:34.041292    5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw"
Nov 03 18:47:34.161172 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:34.161120    5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-nq5bw" podStartSLOduration=8.161104617 podStartE2EDuration="8.161104617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:28.148081029 +0000 UTC m=+4384.158819102" watchObservedRunningTime="2026-11-03 18:47:34.161104617 +0000 UTC m=+4390.171842690"
Nov 03 18:47:34.961261 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:34.961231    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sr54q_1c628272-7a16-4897-af12-59bcfedd74cb/kube-state-metrics/0.log"
Nov 03 18:47:35.382358 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:35.382277    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sr54q_1c628272-7a16-4897-af12-59bcfedd74cb/kube-rbac-proxy-main/0.log"
Nov 03 18:47:35.805627 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:35.805593    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sr54q_1c628272-7a16-4897-af12-59bcfedd74cb/kube-rbac-proxy-self/0.log"
Nov 03 18:47:36.664679 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:36.664654    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-ntxjz_d6bc2a32-53d4-4ad7-a296-74dae0153ded/monitoring-plugin/0.log"
Nov 03 18:47:37.259661 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:37.259628    5838 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxznn_7d4951c9-ab34-4727-9272-fc331a7fae42/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:38.366989 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:38.366959    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9znfw_cb2852e1-7207-4d27-a78e-ab6a06a425c3/node-exporter/0.log"
Nov 03 18:47:38.792309 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:38.792283    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9znfw_cb2852e1-7207-4d27-a78e-ab6a06a425c3/kube-rbac-proxy/0.log"
Nov 03 18:47:39.217936 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:39.217907    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9znfw_cb2852e1-7207-4d27-a78e-ab6a06a425c3/init-textfile/0.log"
Nov 03 18:47:40.338879 ip-10-0-139-156 kubenswrapper[5838]: E1103 18:47:40.338840    5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n7qdx" containerName="extractor"
Nov 03 18:47:40.918841 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:40.918812    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6s4qr_e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/kube-rbac-proxy-main/0.log"
Nov 03 18:47:41.341657 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:41.341577    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6s4qr_e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/kube-rbac-proxy-self/0.log"
Nov 03 18:47:41.765919 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:41.765878    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6s4qr_e3422e3d-3f9a-497d-9eba-50eaa3f7dfc6/openshift-state-metrics/0.log"
Nov 03 18:47:45.167070 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:45.167033    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-q2dlw_c4d0a499-8879-4147-b92b-b81d727ddff3/prometheus-operator/0.log"
Nov 03 18:47:45.588362 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:45.588262    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-q2dlw_c4d0a499-8879-4147-b92b-b81d727ddff3/kube-rbac-proxy/0.log"
Nov 03 18:47:46.012964 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:46.012932    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-tb5x7_c5739357-c9b8-445d-9b36-137395fd0857/prometheus-operator-admission-webhook/0.log"
Nov 03 18:47:46.440478 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:46.440440    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dd9bb94f-kcnnn_99ef5f46-4180-4268-8b66-ae60cb6b6a12/telemeter-client/0.log"
Nov 03 18:47:46.863668 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:46.863594    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dd9bb94f-kcnnn_99ef5f46-4180-4268-8b66-ae60cb6b6a12/reload/0.log"
Nov 03 18:47:47.287662 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:47.287631    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dd9bb94f-kcnnn_99ef5f46-4180-4268-8b66-ae60cb6b6a12/kube-rbac-proxy/0.log"
Nov 03 18:47:47.727211 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:47.727175    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9d7c49b-8dzkd_25a774fe-c887-4a8c-96a6-ed3b863e344c/thanos-query/0.log"
Nov 03 18:47:48.152797 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:48.152766    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9d7c49b-8dzkd_25a774fe-c887-4a8c-96a6-ed3b863e344c/kube-rbac-proxy-web/0.log"
Nov 03 18:47:48.582613 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:48.582509    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9d7c49b-8dzkd_25a774fe-c887-4a8c-96a6-ed3b863e344c/kube-rbac-proxy/0.log"
Nov 03 18:47:49.004641 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:49.004608    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9d7c49b-8dzkd_25a774fe-c887-4a8c-96a6-ed3b863e344c/prom-label-proxy/0.log"
Nov 03 18:47:49.427439 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:49.427407    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9d7c49b-8dzkd_25a774fe-c887-4a8c-96a6-ed3b863e344c/kube-rbac-proxy-rules/0.log"
Nov 03 18:47:49.505444 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:49.505417    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f85rq9ds_c815c4f2-540a-4951-910a-b75c9ad9a72d/istio-proxy/0.log"
Nov 03 18:47:49.848682 ip-10-0-139-156 kubenswrapper[5838]: I1103 18:47:49.848607    5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9d7c49b-8dzkd_25a774fe-c887-4a8c-96a6-ed3b863e344c/kube-rbac-proxy-metrics/0.log"